This provides a brief tutorial on how to run the proactive tagging demo described in the paper. More...

This provides a brief tutorial on how to run the proactive tagging demo described in the paper.

To run the proactive tagging demo, first install all modules as described in the README. Then, adapt the proactiveTagging.xml within icub-client/app/demos to match your system setup (modify the node names). For the role of the individual components, we refer to our paper.

Once you start all modules, two drives will be active as specified in icub-client/app/reactiveLayer/allostaticController/conf/default.ini: one for knowledge acquisition, and another one for knowledge demonstration. Both drives will start decaying (see HomeostaticModule and AllostaticController::updateAllostatic()). Once the drive for knowledge acquisition hits the lower threshold, the AllostaticDrive::triggerBehavior() is being called, which in turn sends an RPC command to the BehaviorManager. Within BehaviorManager::respond(), the Tagging::run() is called.

There, it is specified that the iCub should first find the name of the partner. Therefore, an RPC command is send to proactiveTagging, which where the proactiveTagging::exploreUnknownEntity() method is called. This method uses the face recognition ability of SAM (see SAM.SAM_Drivers.SAMDriver_interaction.SAMDriver_interaction). If the face is not being recognized, it will ask the partner for the name (see proactiveTagging::exploreUnknownEntity() and proactiveTagging::getNameFromSAM()). Finally, the ICubClient::changeName() method is called which changes the name in the working memory and informs other modules of the name change.

The next times the threshold is hit, the iCub will either ask for the name of an object by pointing to it, or ask for the name of the body part by moving it. You can use speech input to provide the name of the object / body part. The possible list of (object, bodypart and agent) names is coded within the XML files in the icub-client/app/proactiveTagging/conf directory. Also, if the name of a body part is known but its corresponding tactile patch is not known yet, the iCub will move the body part and ask you to touch the corresponding part (see proactiveTagging::exploreTactileEntityWithName()). The logic for tagging subsequent objects and body parts is the same as introduced above for naming the partner.

Besides waiting for the drives to decay, you can also give commands to the iCub. The speech commands are recognized within ears::updateModule() that calls the proper module according to the given command. You can say: "Please take the X" (pulling an object), "Give me the X" (pushing an object), "Look at the X", "Point the X", where X is the name of an object. The names are coded in the icub-client/app/ears/conf/MainGrammar.xml file. We recommend keeping these object names consistent with the names in icub-client/app/proactiveTagging/conf. There is some overhead involved when giving commands to the iCub, as every time the Tagging::run() method is being called. This method does nothing in case the object name is known within the working memory, or calls proactiveTagging::searchingEntity() to find which object the human is referring to. If there is only one unknown object, the robot considers that to be the one the human referred to. If there are multiple unknown objects, the iCub will instead try to infer which object is known by recognising a pointing action of the partner.

Once the partner has pointed to the object, the iCub knows the name and can thus execute the requested action. The mapping from command to action is defined in ears::updateModule(). In case an object should be pulled or pushed, the MoveObject::run() method is called. When asked to recognize an action, the RecognitionOrder::run() is called. Finally, the Pointing::run() method takes care of pointing to an object when ordered to do so.

(This page can be edited at src/doc/proactivetagging.dox)