Bartender Bot
Fundamentals of robotics
Skills Advanced: Forward & Inverse Kinematics, Path Planning & Trajectory Generation, Computer Vision, Python, Robotics
Objective: Utilize the Kinova Gen3 Lite robot arm, along with an Intel RealSense camera, to detect a can and a cup using fiducial markers, pick up the can, pour its contents into the cup, and set it back down
In this project, I worked with a team of 3 on the complex manipulation task of pouring drinks with a robotic arm. A container of fluid that is not kept upright throughout the trajectory will spill in undesired locations, which can damage the operating robot and is a nuisance to clean up. This sort of task also has clear applications in assistive tech, where people with limited mobility could use assistance with activities requiring fine motor skills like pouring.
The platform used for this project is the Kinova Gen3 Lite, a 6-degree-of-freedom (DoF) robot arm, which means it can reach any pose for any position within the working area of the arm. The coordinate frame assignments and corresponding Denavit-Hartenberg (DH) parameters are shown below.
Fiducial marker recognition
The pose we cared most about for manipulation was the end-effector (EE). Since we controlled the joint angles, we needed a way to map them to the EE pose. We did this through forward position kinematics by converting the DH parameters into homogeneous transformation matrices and using matrix multiplication to combine the transformations to reach the EE pose.
Next, we focused on getting the required joint angles for a desired EE pose, also known as inverse kinematics. We chose to use a numerical method. Although it is more computationally intensive and not guaranteed to converge, it is generally simpler and faster to implement because it does not require a unique derivation for each robot platform, unlike analytical methods. We chose the Newton-Raphson method for our numerical inverse kinematics, which utilizes a matrix known as the inverse Jacobian to iteratively optimize joint angles until they result in the desired EE pose.
The Jacobian is a matrix that describes how EE motion relates to joint rotation. It is broken into linear and angular components, shown to the right. We also implemented the damped-inverse method to shift the Jacobian away from singularity conditions where the joints converged to a position that is impossible for the calculation to escape from, also shown on the right. Finally, the Jacobian can be inverted and used to iteratively update joint angles toward a desired EE pose.
Checkerboard camera calibration to account for fisheye effects
Can jig and marker
Program block diagram
End-Effector Pose
Forward Position Kinematics
Joint Angles
Inverse Kinematics
Calculation of the Jacobian
Damped inverse method to avoid singularities
Newton-Raphson Method to find joint angles for desired EE pose
To maximize the chance of the numerical inverse kinematics converging to a valid set of joint angles, we generated 200 random starting configurations and let each run for a max of 300 iterations. After each iteration, we used forward kinematics to calculate the error between the current EE position and the desired one, and either accepted the joint angles if the error was low enough to meet our threshold or ran another iteration of Newton-Raphson if it was not.
Another very valuable section of the project that I was not responsible for was the computer vision. We used an ArUco marker library with known marker IDs and sizes to calibrate a depth camera and get the transformation from the camera frame to the world/robot base frame. For the cans and cup, I created 3D-printed jigs that gave us a consistent offset from the marker to the object while ensuring that the marker was fully visible to the camera. The computer vision allows the cans to exist at any point within the operating range of the robot and still be picked up without any adjustments to the code, as demonstrated in the videos below.
Putting it all together, this gave us a way to calibrate where the robot was relative to the cups and cans it would manipulate and a way to control the EE to move to object locations and grasp them. To keep the cans upright, we simply set constant EE rotation angles that kept the cans vertical for every point on the path that was not a pouring step. This can be observed in the videos below.