A module combining multiple libraries and modules from the iCub repository that allows to execute some basic and complex actions on objects placed on a table in front of the robot.
Collaboration diagram for actionsRenderingEngine:
A module combining multiple libraries and modules from the iCub repository that allows to execute some basic and complex actions on objects placed on a table in front of the robot.
Description
This module makes use of the actionPrimitives library in order to interact with the objects placed on a table. The module can independently receive visual inputs from a stereo/mono tracker or from an object detector. The module will ask the system to perform the translation from stereo to cartesian information according to the current "stereo to cartesian" modality chosen:
– homography: which assumes that the target point lies on a table and exploits homography to obtain the relative 3D cartesian coordinates.
– disparity: which uses the StereoDisparity module to obtain the 3D cartesian estimate of the 2D target.
– network: which uses a previously trained neural network structure to predict the 3D cartesian coordinate from the stereo input.
Issuing commands
The commands sent as bottles to the module port /<modName>/cmd:io are described in the following. The response to any command consists in the vocab [ack]/[nack] in case of success/failure.
Some commands require to specify a visual target for the action required. In these cases the parameter [target] can be expressed as follows:
(notation: [.] identifies a vocab, "." identifies a string, <.> identifies a double)
–[motion] when using motion cues to detect the visual target.
–[track] when using the output of the visual tracker.
–[fixation] when the target is assumed to be in both cameras image centers.
–[raw] when the target is provided as a raw couple of camera plane coordinates to one of the two 'raw:i' ports.
–("left"|"right" u v) when the target is a 2D couple of camera plane coordinates. If not specified, camera is assumed to be "left".
–("cartesian" x y z) or (x y z) when the target is a 3D cartesian position wrt the robot's reference frame.
–("cartesian" x y z ax ay az theta) or (x y z ax ay az theta): occasionally also the orientation can be provided.
–"object_name" when using a visual classifier to localize objects. This option represents a label to access online information stored in the databased provided by objectsPropertiesCollector module. The label itself is used to access the corresponding object through the "name" property, whereas the relevant 3D information are available through the "position_3d" property.
General Parameters:
The majority of commands can usually be provided with general optional parameters:
–<arm>: the user can specify the arm required for the action ("left"/"right"). Otherwise the default arm will be used.
–"no_head"/"no_gaze": the action does not affect the gaze of the robot (this parameter does not influece commands that explicitly require to move the head like LOOK).
–"no_sacc": disables the saccadic movements of the gaze controller during the action.
–"still": prevents the robot from bringing back home the arm after having accomplished the action.
The commands:
IDLE
format: [idle]
action: the gaze controller is reset to the original context stored at start and head/eye control is interrupted.
HOME
format: [home] "param1" "param2"
action: the arms are sent to home position. the optional parameters "param1" and "param2" can be independently set to "gaze" or "head" in order to bring the gaze controller in home position and "fingers" or "hands" to open the robot hands. Alternatively the parameter "all" can be supplied to perform both optional actions.
OBSERVE
format: [observe]
action: if the robot is holding an object it brings it in the FoV of its cameras with the final purpose of visually explore it.
DROP
format: [drop] "param1" or [drop] "over" [target] "param1"
action: if the robot is holding an object it brings it over the table and drops it on a random position approximatively in front of it. Optional parameter "away" can be provided in orded to have the robot leave the object outside the cameras FoV (on the right or left side of the robot depending on the arm in use). Optional parameter "over" can be supplied together with the action [target] in order to have the robot drop the object on top of the visual target.
HAND
format: [hand] "action-type" "hand-type"
action: perform the hand sequence specified by the action-type with the hand specified by the hand-type.
TAKE
format: [take] [target] "param1"
action: the robot tries to reach the specified [target] and grasp it. Optional parameter "side" or "above" can be supplied to choose the orientation the robot should try to mantain while performing the action (default: "above").
TAKE_TOOL
format: [tato] "param1"
action: the robot will reach a specified position to take the tool from a user. Optional parameter "left" or "right" can be supplied to choose the orientation the robot
GRASP
format: [grasp] [target]
action: the robot tries to reach the specified [target] and performs a power grasp. The target must be specified both in cartesian position and orientation.
As further parameter user may specify the way the robot will approach the target by providing the options ("approach" (dx dy dz wrist_pitch)), where dx/dy/dz account for offset displacement in meters wrt to the grasping reference frame and wrist_pitch is the apporaching pitch of the wrist given in degrees.
TOUCH
format: [touch] [target] "param1" "param2"
action: the robot tries to reach the specified [target] and then brings the arm back to home position. Optional parameter "side" or "above" can be supplied to choose the orientation the robot should try to mantain while performing the action (default: "above").
PUSH
format: [push] [target] "param1"
action: the robot tries to reach the specified [target] from one side and then push it laterally. Optional parameter "away" can be supplied in order to have the robot push the object away from its root reference frame.
POINT
format: [point] [target]
action: the robot tries to point at the specified [target] with its index finger.
POINT_FAR
format: [pfar] [target]
action: the robot tries to point at the specified [target] located far from the robot.
LOOK
format: [look] [target] "param1" (block_eyes <ver>)
action: the robot looks at the specified [target]. "param1" can be set equal to "fixate" in order to keep the gaze fixating the requested target also when other commands are issued to the torso. The further option "wait" can be specified in order to let the robot wait until the target is under fixation before returning.
If provided, the option (block_eyes <ver>) serves to block the eyes at the specified vergence while gazing.
Note: the special target [hand] (with optional parameter "left"/"right") can be provided to have the robot look at its own hand. The robot will keep looking at its own hand until an idle command.
EXPECT
format: [expect]
action: the robot puts one arm forward with the palm of the hand facing up and waiting for an object to be put on it.
GIVE
format: [give]
action: the robot puts one arm forward with the palm of the hand facing up and opens the fingers so that the object held in the hand is free to be taken.
TRACK
format: [track] [target] "param1"
action: the specified [target] position is supplied to the tracker module and the gaze controller is updated in order to keep the object constantly inside the robot fovea. The optional "param1" can be put equal to "no_sacc" in order to disable the saccadic movements of the gaze controller during the tracking process.
TEACH ACTION
format: [teach] "action_name" [start/stop] "param1"
action: If parameter [start] is specified and the action "action_name" has not been registered yet, the robot arm is set to torque control mode. The cartesian position (wrt the starting point) and orientation of the robot arm is stored until the command [teach] "action_name" [stop] is received. The learned action is recorded to the file "actions/<arm>/<action_name>.txt" and can be repeated autonomously using the command IMITATE ACTION ([imitate] "action_name").
IMITATE ACTION
format: [imitate] "action_name"
action: the system loads the file "actions/<arm>/<action_name>.txt" and controls the arm in order to repeat the trajectory previously registered.
CALIBRATION
different types of calibrations can be requested by issuing commands of the form: [calib] [calibration_type] "param1" "param2" In the following a short description of the possible values of [calibration_type]:
–[table] the robot will try to find out the table height exploiting the contact detection based on force control. If the contact is detected then the new table height is sent to the homography module that is in charge of extracting the 3d coordinates of the object from its 2d projection on the image plane. The file containing the table information is also updated accordingly to avoid calibrating always at start-up.
–[fingers] the robot will calibrate the fingers in order to detect whether correct grasp has been achieved.
–[kinematics] this option requires a parameter [start/stop]. when started, the system is set to cartesian torque control and the robot arm can be moved around by the human user. When the [calib] [kinematics] [stop] command is received, the system is set to stiff mode and the offset between the initial and the current position is stored. It is also possible to associate a specific kinematic offset to a single object. To do so, it is required to issue the command [calib] [kinematics] [stop] <object_name>. Then, whenever the system will be asked to preform an action over such object, the system will use the learnt offset.
EXPLORATION
different types of explorations can be requested by issuing commands of the form: [explore] [exploration_type] In the following a short description of the possible values of [exploration_type]:
–[torso] the robot will start moving its torso exploring different positions (specified in the file exploration_poses.ini). then it will go back to the initial position.
–[hand] the robot will start moving its hand (typically while holding an object) and at the same time look at it. For safety issues, by default the robot moves the other hand back to home position in order to avoid collision. This behaviour can be modified by providing the optional parameter "keep_other_hand_still".
The "Get" Port
The actionsRenderingEngine can be queried via the port /<modName>/get:io in order to obtain the following data:
GET format: [get] <request> action: the system returns the requested element.
<request> can be of the form:
–[s2c] [target_1] ... [target_n]: returns the current cartesian coordinates of the n visual targets wrt to the robot root reference frame. The number of targets is not predefined.
–[table]: returns the current height of the table in front of the robot
–[holding]: returns true if the robot is holding an object in its active hand.
–[hand] [image]: returns a vector of the form (u_left v_left u_right v_right hand-head-distance) that reports the projection of the end-effector of the active arm in both the robot cameras together with the distance (in meters) between the robot forehead and its hand.
Libraries
- YARP libraries.
- actionPrimitives library.
Ports Accessed
Assume that the robot interface is operative and the wholeBodyDynamics is running.
Ports Created
Aside from the internal ports created by actionPrimitives library, we also have:
- / <modName>/wbd:rpc to be connected to /wholeBodyDynamics/rpc:i in order to allow for resetting force offsets in the computation of internal dynamics of the robot.
- / <modName>/cmd:io receives a bottle containing commands whose formats are specified in the previous section. The port replies as soon as the current action has been completed.
- / <modName>/rpc remote procedure call. Recognized remote commands:
-[help]: returns the list of available rpc commands.
-[get]: get requests
-[status]: returns the bottle (gaze <status>) (left_arm <status>) (right_arm <status>) where <status> can be equal to "idle", "busy" or "unavailable".
-[impedance] [on]/[off]: enable/disable (if available) impedance control.
-[waveing] [on]/[off]: enable/disable the iCub arm(s) waving.
-[mode] [homography]/[disparity]/[network]: sets the desired stereo to cartesian mode.
-[interrupt]: interrupts any action deleting also the action queue for both arms.
-[reinstate]: if the module was interrupted reinstate it.
-[elbow] "left"|"right"|"both" <height> <weight>: to change elbow parameters.
-[time] <time>: to change default arm movement execution time (in seconds).
Parameters
The following are the options that are not usually contained within the configuration file.
–name name
- specify the module name, which is actionsRenderingEngine by default.
–from file
- specify the configuration file (use –context option to select the current context)
Input Data Files
– kinematic_offsets.ini contains the table height as well as the arms kinematic offsets and is updated on-line as result of an exploration phase. – network.ini contains the information needed by the system to load the neural network that computes the stereo2cartesian mapping. – hands_sequences.ini contains the list of hand sequences for the action primitives. – exploration_poses.ini contains a list of position used to explore multiple points of view with the robot torso. – grasp_model_<hand>.ini contains the information needed by the PerceptiveModel layer. – actions/<arm>/<action_name>.actions each action "learned by imitation" is stored in a .txt file with the same name of the action itself.
Tested OS
Windows, Linux
