Metafinger
This page introduces the Metafinger, a soft robotic finger capable of multimodal proprioception. The Metafinger is a specific design of Soft Polyherial Networks[1, 2, 3, 4, 5, 6, 7]. The core components are the soft finger and the camera under it. We also installed a dev board with the camera, which can capture the images and send out via WiFi or USB. The metafinger is deployed at the end of the beam on MagiClaw and estimates the force and shape during interaction.
Preview
Specifications
Module
LicheeRV Nano W
CPU
SOPHGO SG2002; Major core: 1GHz RISC-V C906 / ARM A53 (selectable); Minor core: 700MHz RISC-V C906; Low-power core: 25～300M 8051
NPU
1TOPS INT8, supports BF16
Memory
Integrated 2Gbit (256MByte) DDR3
Wi-Fi
Dual-band 802.11ax
Bluetooth
Bluetooth 5.4/ BLE
Storage
1 x microSD card slot
USB
1 x USB 2.0 OTG Type-C
Camera
GC4653
Lens
1.8mm 1/4" 5MP
Frame rate
60fps@720P (max)
Dimensions (W x D x H)
40mm * 40mm * 115mm
Weight
56g
Operating Temperature
0 ~ 45℃
Flash Image
The finger dev board, LicheeRV Nano W, needs a TF card with system image. Please follow the following steps to flash the image.
Find the latest system image file on Google Drive, for example, metafinger-latest.img.
Download balenaEtcher (highly recommended), install and launch it.
Insert a TF card into the card reader and connect it to your PC. In balenaEtcher, select the downloaded image file, choose the TF card, and click Flash
Insert the TF card into the dev board, power it on, and wait for the system to boot. The first boot may take a little longer, just be patient.
Connect to Dev Board via SSH
Since the WiFi settings are not configured, you need to connect the dev board to the PC via a USB cable, and the board will emulate as a USB network adapter. Then the board and the PC will be on the same local network through the USB connection. By default, there are two types of USB virtual network adapter drivers (NCM and RNDIS drivers) to meet the needs of different systems.
All Windows systems will automatically install the RNDIS driver, while only Windows 11 will automatically install the NCM driver. As long as one of the drivers works, it is sufficient.
Open Task Manager -> Performance, and you should see a virtual Ethernet with an IP address such as 10.131.167.100, which is the computer's IP address. The device's IP address is the same but with the last digit changed to 1, i.e., 10.131.167.1.
If you are using Windows 11, you will see two virtual network adapters; you can use either IP address.
Use ifconfig or ip addr to see the usb0 and usb1 network interfaces, and either IP address can be used. The IP address you see, such as 10.131.167.100, is the computer's IP address, and the device's IP address is the same but with the last digit changed to 1, i.e., 10.131.167.1.
Check for the usb network adapter under System Settings -> Network. The IP address you see, such as 10.131.167.100, is the computer's IP address, and the device's IP address is the same but with the last digit changed to 1, i.e., 10.131.167.1.
To configure the connection settings on the PC, open Terminal and run
# SSH by IP
ssh root@<ip address>
# SSH by hostname
ssh root@<hostname>.localwhere root is the default user name, <ip address> is the IP address of the dev board which can be found on the connected PC, and <hostname> is the hostname can be found in the dev board. Both of them are capable, just select your preferred way. Type yes for the following message
ECDSA key fingerprint is SHA256:XXXXXXX.
Are you sure you want to continue connecting (yes/no/[fingerprint])?Then enter the default password root. If you have successfully connected to the dev board, you will see the following output
Configure WiFi Settings
After connecting to the dev board via SSH, run the following command to set the WiFi:
python /root/wifi_connect.py --ssid <ssid> --pwd <pwd>where <ssid> is the name of the WiFi, and <pwd> is the password. After connecting to the WiFi successfully, the output will show the IP address of the dev board obtained from the router.
Assemble Camera and Lens
GC4653 is a MIPI camera with good image quality. It can achieve a high frame rate of 60fps at 720p resolution. But FOV of the original lens is not enough for the Metafinger, so another wide-angle lens is selected to replace it. Please follow the steps to assemble the camera and lens.
Remove the original lens from the camera and replace it with the provided wide-angle lens.
Install the camera onto the MIPI camera mount, paying attention to the wire sequence. Then power it on, and wait for the system to boot.
Clone the latest magiclaw repository and install all dependencies on either the control box or your PC:
git clone https://github.com/asMagiClaw/magiclaw.git
cd magiclaw
pip install -e .Test whether the images captured by the camera can be streamed successfully:
python scripts/read_camera.py --addr <addr>where <addr> is the IP address of the dev board. A window will show streaming images from the dev board.
Rotate the lens to adjust the focal length so that the camera can capture the marker clearly.
Calculate Camera Parameters
To obtain the marker's pose, we need to calibrate the camera's intrinsic matrix and distortion coefficients. We recommend using a black-white chessboard (12 x 9, square size of 1.5mm).
Power the dev board on, and prepare the chessboard ready.
Collect chessboard images streamed from the dev board:
python scripts/calibrate_camera.py --name <name> --addr <addr>where <name> is the camera name defined for saving directory, and <addr> is the IP address of the dev board. A window will show streaming images from the dev board.
Press c to capture the image, and ESC to quit capturing. All images will be saved under ./data/camera_calibration/<name>_<width>x<height>/ directory, where <name> refers to the name defined above, and <width> and <height> refers to the size of the captured images.
After pressing ESC, you will be asked which software is to used for calculation, either OpenCV or MATLAB. To use Open CV, enter y, and the script will start calibration automatically. To use MATLAB, enter n, and the script will exit.
If using MATLAB, first launch the MATLAB and open the Camera Calibrator app. Press Add Images and select the saved images. Enter 1.5 for Size of checkerboard square in a new window Image and Pattern Properties. Open Options -> Radial Distortion, and select 3 Coefficients. Press Calibrate to start calibration.
Export camera parameters and a variable will be created containing calculated results. A MATLAB function is provided in scripts/calib2yaml.m to export the parameters to a .yaml file. Running following command in the MATLAB command line:
cd scripts
calib2yaml(<variable_name>, <file_name>)where <variable_name> refers to the name of the variable containing results, and <file_name> refers to the name of the .yaml file for saving parameters. The file will be saved under the same scripts/ directory.
Modify the .yaml file for camera parameters. A file example is provided as configs_example/camera/maxicam-xxxx.yaml, containing mode, host, width, height and so on. You need to replace the values of dist and mtx with the calculated results. This file should be saved under configs/camera directory, named as hostname.yaml consistent with the hostname of the dev board.
Marker Tracking
Metafinger uses ArUco markers for pose tracking. The camera captures the images of the marker fixed on the soft finger and streams to the control box or the PC. Then the marker is detected and the pose is estimated using OpenCV. The value change of the pose represents the deformation of the soft finger and will be as the input of the model to infer the force and shape.
Model Training
The soft finger structure can "feel" the force and own shape, called as "proprioception", during physical interaction with environment. To estimate the force and shape, we provided a proprioceptive learning framework for the soft finger. Please follow the guide in metafinger repository to train and test the model.
Bill of Materials
Soft finger
Hei-Cast 8400
$ 25.00
Finger interface
4 cm wide, 4 mm thick with felt
$ 2.00
CAD Models
STEP files of all 3D-printed parts are provided on Google Drive.
Dev Board Resources
Datasheets, schematic diagrams, drawings, and more about the dev board can be found here.
Last updated