
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
%        1         2         3         4         5         6         7         8

\documentclass[letterpaper, 10 pt, conference]{ieeeconf}  % Comment this line out
                                                          % if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf}      % Use this line for a4
                                                          % paper

\IEEEoverridecommandlockouts                              % This command is only
                                                          % needed if you want to
                                                          % use the \thanks command
\overrideIEEEmargins
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document



% The following packages can be found on http:\\www.ctan.org
%\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
%\usepackage{amsmath} % assumes amsmath package installed
%\usepackage{amssymb}  % assumes amsmath package installed

\title{\LARGE \bf Robot Google$^{TM}$: a Framework for Robot Knowledge Sharing}

\author{\authorblockN{Xiuyi Fan and Thomas C. Henderson}
\authorblockA{School of Computing\\
 University of Utah\\
Salt Lake City, UT, 84112 USA\\
Email: \{fan, tch\}@cs.utah.edu}
}


\begin{document}

\maketitle
\thispagestyle{empty}
\pagestyle{empty}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
Knowledge representation is a traditional field in artificial
intelligence. Researchers have developed various ways to represent and
share information among intelligent agents. Agents that share
resources, data, information, and knowledge perform better than agents
working alone. However, previous research also reveals that sharing
knowledge among a large number of entities in an open environment is a
problem yet to be solved. Intelligent robots are designed and produced
by different manufactures. They have various physical attributes, use
different knowledge representations and have different needs. In this
research, we pose robot knowledge sharing as an activity to be
developed in an open environment - the World Wide Web. Just as search
engines like Google provide enormous power for information exchange
and sharing for humans, we believe a searching mechanism designed for
intelligent agents can provide a robust approach for sharing knowledge
among robots. We have developed knowledge representation for
robots that allows Internet access and a knowledge organization and
search indexing engine that performs knowledge retrieval.
\end{abstract}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{INTRODUCTION}
In the past when we needed to know something, we would look it up in
an encyclopedia or find a book on the subject.  Nowadays, we turn to
web search engines, like Google$^{TM}$\footnote{GOOGLE is a trademark of Google Inc.}
or Yahoo$^{TM}$\footnote{Yahoo is a trademark of Yahoo Inc.}, and are given
pointers to a large amount of information, and we usually find what
we're looking for relatively quickly and easily. In this research, we develop
similar capabilities for physical robots, including humanoid robots,
which act in the world and must know a great deal about it.  This
includes robot butlers, surgeons, drivers, hospital orderlies,
homecare nurses, etc. Thus, when a robot encounters an unfamiliar or
unknown object in its environment, or when it needs to know how to
perform a particular task with or on an object (e.g., clean it), it
will be able to query a Robot Google in order to get pointers
to relevant information available in the world wide web.

Humans achieve knowledge sharing mainly through natural language: queries
are words that are matched to document content.  For robots, it is not
clear how to achieve this, and the question arises as to what
representations best facilitate robot knowledge sharing. It is quite
clear though, both the content of the knowledge representation and its
format impact the effectiveness of this sharing. We would like to build
our framework on solid ground so that knowledge representations
derived from it allow common usage. Our framework 
provides a convenient yet unambiguous way to support the
representation. Robots that use our framework may communicate with
each other usefully.

\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/fig1.eps,width=2.6in}
    }
  }
  \caption{The Robot Google Framework.}
  \label{Figure:image1}
\end{figure}

The developed framework for our solution is shown in
Figure~\ref{Figure:image1}. In this figure, each participant robot
creates web accessible knowledge repositories; the Robot Google server
harvests knowledge from each of the participant robots and then
creates and organizes efficient indexes into the database. Participant
robots query the Robot Google server for knowledge and receive URLs
pointing to other robots' knowledge. Robots, in this view, act as
agents \cite{Ferber,Maes,Subrahmanian,Weiss,Wooldridge00},
and we assume their ability to generate the necessary 
knowledge structures; this is not an issue of investigation here.

Imagine a scenario like the following. A kitchen robot works in a
kitchen. It is told to clean all kitchenware. After successfully
cleaning a few plates, forks and spoons, the robot notices that there
is a pair of wooden sticks. The robot is confused by this pair of
wooden sticks as it has never seen this before. It does not know what
to do with them. By asking a nearby human, the robot learns that this
pair of wooden sticks is called ``chopsticks.'' The robot then
formulates a query and sends it to the Robot Google server though its
on-board Internet connection. It wants to know if these chopsticks
need to be cleaned (as it is told to clean all kitchenware) and if so,
how to clean them. The Robot Google server processes the query, and
responds with: please go to www.robot\_chopstick.com/info to see more
information about chopsticks and please go to
www.robot\_chopstick.com/clean to see how they can be cleaned. The
robot then access these URLs and downloads 
the needed information. It then cleans the pair
of chopsticks successfully.

As a general framework, we propose to consider a robot to be much like
its human counterpart sitting at a computer terminal in the following
sense.  Two major paradigms have been established to support people:
\begin{enumerate}
  \item
  Java virtual machine

  \item
  Web-based search engines.
\end{enumerate}
The first provides a means for a particular machine to emulate a
universally shared virtual machine, and in effect, to embody another
machine through the execution of byte-code.  An analogous approach for
robots requires the definition and adoption of a universal reference
standard for some abstract mechanism, and the local embodiment of that
abstract machine in the particular instance of a specific robot (e.g.,
executing bot-code).  There could be a variety of abstract reference
machines; e.g., mobile 3-DOF vehicles, 6-DOF robot arm mechnisms, or
44-DOF humanoid linkage mechanisms (e.g., see~\cite{Knoop2007}).  Each
robot manufacturer would then provide interpreters for such abstract
reference models.

The second paradigm, web-based search engines, addresses ways to find
and share relevant knowledge.  We describe in this paper a specific
framework in which all queries and information gathering flow through
the search engine (see Figure~\ref{Figure:image1}).  This follows
human use of a search engine and allows knowledge indexing, querying
and relevance determination (the latter is possible since requests for
URLs must also be submitted through the Robot Google search engine).

A large amount of work need to be done to realize the Robor Google
scenario.  Questions need to be answered include: (1) How does a
robot communicate with Robot Google? (2) How does Robot Google process
and answer a query? (3) How does one robot know if information stored
on a website applies to its own environment? These are interesting
questions that point to intensive research. This work focuses on
solving one core problem: build a multi-format data search engine for
robot knowledge sharing.

The next section describes some relevant background work.  Section III
presents our view of robot knowledge. Section IV introduces the Robot
Google architecture.  We discuss functions and designs of Robot Google
components.  Section V presents experimental results on our sample
data set. We conclude in section VI with future research. 

\section{RELEVANT WORK}

The study of knowledge representation can be traced back to ancient
Greece. Epistemology, the study of the nature of knowledge and its
justification, was established by Plato in the fifth century
B.C. \cite{Sowa00}. Since then, the study of knowledge, including its
nature, representation, development, etc. has been carried on by
philosophers, mathematicians, linguists, and scientists. Most
knowledge representation developed today is rooted in various
logics. Recently, some computer scientists have expressed belief that
grounding knowledge purely in logic, e.g., in symbolic languages, is
insufficient for building intelligent agents, e.g., robots. They
propose to develop sensor grounded and context-aware knowledge
representations for robot \cite{Roy05b, Roy05a}. Even though their
work is promising, they are still far from providing a comprehensive
and satisfactory solution.

Although still in its formative stages, several groups are making
progress on sensor-grounded robot knowledge creation. Cohen et
al. \cite{Cohen2000} describe a {\em natural semantics} approach in
which robots learn meanings through their interaction with the
environment. In natural semantics, meanings are acquired and
maintained by the robot system, and not specified externally by human
programmers or knowledge engineers.  In this work, a robot is provided
with a small number of behaviors (e.g., move, turn, open gripper,
etc.), and the robot records sensor data streams. In this way, the
robot learns a sensor data based ontology through interaction with the
environment, and concepts are related to the sense data. 

The Spatial Semantic Hierarchy, which allows bootstrap learning from
uninterpreted experience \cite{Kuipers2001}, the human developmental
theory based robot behavior study \cite{Grupen2005}, the relational
representation for procedural task knowledge \cite{Hart2005},
the cognitive robot \cite{Dillmann,Dillmann2005}, etc, are some examples
of a group producing sharable robot knowledge.

Another influential work in knowledge sharing in general is the
Knowledge Interchange Format, known as KIF \cite{Genesereth}. KIF was
defined as an ANSI standard by the NCITS T2 committee on Information
Interchange and Interpretation in 1998. KIF is a version of typed
predicate logic, which differs from the sensor data grounded knowledge
sharing. The book by Davies et al. \cite{Davies03} provides a very clear
review of methods and tools developed for the human semantic web,
which set its aim as to create a universal medium for information
exchange by putting documents with computer processable meaning on the
World Wide Web. Using the Semantic Web, information can be better
organized and more accurately delivered to a human reader.

\section{ROBOT KNOWLEDGE}
Humans recognize the external world first through sensory
organs. Imagine when we visit a museum,
there are a number of artifacts on display. Suppose there is an object
we do not recognize, but we would like to know what it is. We
look at it to see its shape; we lift it to feel its weight; we may
smell it to determine its odor; we may tap it with our finger to see
how it sounds.  With this collected sensory information, we try to
associate this new object to some object we already know.  We believe
that humans recognize an object first by collecting information
through sensory organs.  Sensory information is the ground for
object recognition.

We believe robots can behave similarly, and that the best
way for a robot to recognize objects is through its sensor
data.  Therefore, we position this research towards a sensor data
grounded approach.  We restrict the scope of robot knowledge to be:
string representation of sensor data.  Strings give information
about the object; this usually includes physical properties of an
object and verbal descriptions. For a robot to know an object means to
have information about that object stored in its memory.

The problem we need to address is: how do robots share their
knowledge with each other, though a knowledge server such as Robot
Google?  We  believe two knowledge transformations can help.
The first transformation takes place in robots, where knowledge is
transformed from the robot's internal representations, which are
probably known only to themselves, to a form such that they are
understandable to other parties, i.e., Robot Google and other
robots. The second transformation takes place in Robot Google, where
knowledge which is represented in the format produced by the first
transformation, is transformed into a representation, that can be
efficiently indexed.  Then, Robot Google is able to effectively answer
queries sent by other robots, hence knowledge is shared. 

\subsection{The First Transformation}
The purpose of the first transformation, from a robot's internal
format to an open standard, is to transform knowledge in a
systematic way such that an unambiguous, widely-adoptable, format is
obtained, while maintaining all information in a {\it knowledge
piece}. Two requirements need to be satisfied.  First, the
data source, or inputs, of the transformation need to be collected
in a standard way.  We propose the idea of the {\it asymmetric
spatial-temporal coherence} for objects.  When a robot collects
information about an
object, i.e., measures its properties, we assume the robot does this
in a uniform way such that all properties are measured with the least
intervention among them. Once all
information is collected through a robot's onboard sensors, the
robot needs to package them tightly to maintain data
integrity. Therefore, it is clear to both the robot, and
Robot Google, at a later time, that information about one
particular object, or an instance of an object, is collected, not just
information about some object from a certain class.

Having a clear distinction between an object instance and an
object class is significant to this work for two reasons. First,
Robot Google is deeply rooted in the concept of sensor data grounded
knowledge. Knowing which instance sensor data refers to is important
to all possible higher levels, 
e.g., semantic level, knowledge structure formation at a later
time. Second, it is desirable to support instance-based queries in
addition to the general class-based queries.  For example, to be able
to determine that an image represents a human face is useful (the
class-based query), but to be able to detect whose face it is
(instance-based query) can be more useful for some applications.

We employ the standard Extensible Markup Language (XML) to
represent knowledge as the result of the first transformation, as XML
is widely used and accessible. The designed format is flexible
enough to capture various types of knowledge while still being parser
friendly. 

\begin{table}
\begin{tabular}{|c|c|}
\hline
{\bf Field}&{\bf Value}\\
\hline
\hline
IM1:red & [1.84e-011 -6.63e-009 4.44e-007 3.07e-005 -4.80e-004] \\
IM1:green & [2.23e-011 -8.72e-009 8.06e-007 8.99e-006 -2.25e-004] \\
IM1:blue & [2.20e-011 -8.55e-009 7.65e-007 1.37e-005 -4.56e-004] \\
IM2:red & [1.45e-012 2.51e-009 -9.89e-007 8.74e-005 -2.45e-004] \\
IM2:green & [3.11e-012 1.82e-009 -9.20e-007 8.84e-005 -3.80e-004]\\
IM2:blue & [4.23e-012 1.34e-009 -8.71e-007 8.94e-005 -5.51e-004]\\
IM1:edge & [-1.24e-010 6.39e-008 -1.07e-005 6.47e-004 -0.0063] \\ 
IM2:edge & [-1.37e-010 6.99e-008 -1.16e-005 6.97e-004 -0.0070] \\
LSI: & [-0.0509 0.2674 0.2571 0.4403] \\
Dim: & [6.0190 1.6906 11.3570 0.4998] \\
Weight: & 0.4244 \\
Filename: & 'knife2a.jpg' \\
Description: & 'Knife with black handle' \\
\hline
\end{tabular}
\caption{An Object Example Contains Images, Text Descriptions,
  Dimensional Data and Weight Measure}
\label{Table:obj_exp}
\end{table}

\subsection{The Second Transformation}
The purpose of the second transformation is to convert the
easy-to-communicate XML file into something
that is easy to index. Hence we can build the search engine
efficiently. We take the vector space approach.

Every knowledge piece in our system can be divided into three parts:
text data, sensor data and meta data. Text data are provided by
humans. This includes the name, function, use and possible other related
descriptions of an object. Sensor data represents physical
properties of an object. They are recorded by numerical values. For
instance, the weight measure of an object is recorded with a single
numerical value, given a standard unit is used; the shape of an
object can be recorded by a histogram of the direction of the object's
edges, where a histogram is represented by a vector. Meta data is
recorded when the object is measured by sensors. It contains
information about collected sensor data. For instance, the location of
where the object is encountered, the time of when the object is
encountered, the type/band/model of the sensor used to collect this
data, etc.

Knowing the curse of dimensionality\cite{Castelli2002}, we
explored a few methods to reduce lengths of vectors, i.e., convert a
high dimension vector to a lower dimension one, into especially
histograms, including, Fourier coefficient representation, polynomial
coefficient representation, statistics representation and moment
representations.  Our results show that representing a $256$-bin color
histogram by coefficients of a fourth order polynomial provides a
reasonable compromise between data accuracy and vector length
reduction; hence it is adopted in this research. Text information is
converted into vector space using the latent semantic indexing (LSI)
mechanism \cite{sclaroff99, deerwester90}. For instance, a description
such as ``Small metal bowl'' can be represented as $[-0.39, -0.12, -0.21, 
0.08]$. Meta data can be represented by numerical values as well. For
instance, time can be unambiguously represented by a UNIX time string;
locations can be represented by GPS coordinates and sensor type can
be represented by an index of the sensor in a sensor database or
Logical Sensor System \cite{Henderson98b,
  Henderson84a}. Table~\ref{Table:obj_exp} shows an example of an 
object represented in this vector space.

\section{KNOWLEDGE SEARCH ENGINE}
As described by Frieden and Kuntz \cite{Fielden2002}, the three main
tasks of a search engine are to (1) match query keywords with related
material on the web, (2) rank web documents according to relevance,
and (3) provide pointers to the documents. Essentially, a search
engine needs to be able to gather data from online sources, categorize
and store them into a repository, and provide data access to its
users. In the first generation robot search engine, i.e., Robot
Google, we do not foresee a major role for web crawlers.  Even if web
pages exist, the meta data is not available to determine what pages to
download, what is of interest in them (e.g., there are no words to
count and no lexicon to help define any semantics), no popularity
measure, and no standard places to find things (e.g., specific sites,
in homepage, etc.).  Thus, robots must register with the system
and provide direct meta data and links to files. We focus on creating
a robot search engine architecture that is: 
\begin{itemize}

  \item
  {\bf Flexible.} Robot Google is not designed for any particular data
  type.  We emphasize the {\it sensor data grounded} knowledge sharing
  model, in which virtually all types of sensor data can be
  supported. 

  \item
  {\bf Scalable.} Robot Google sets no limit on the number of
  attributes or properties of its supported data types. Both query
  templates and items stored in Robot Google can contain as many
  properties as needed.

  \item
  {\bf Efficient.} Robot Google supports indexing on a large amount of
  data. The retrieval speed is not determined by the number of items
  stored in its database.

\end{itemize}
To accomplish these three tasks, we divide Robot Google into four
groups of components: a query processor, indexing structures, a cross
analyzer and a response formalizer (see
Fig~\ref{Figure:RG_architecture}).
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/RG_Arichtecture.eps,width=3.0in}
    }
  }
  \caption{The Robot Google Architecture.}
  \label{Figure:RG_architecture}
\end{figure}

\subsection{Query Processor}
Robot Google supports query-by-example \cite{Castelli2002}.
The query processor is the first component in a Robot Google query
workflow. It takes queries, in the format of XML, and translates them
into arrays of vectors and sends each vector to a corresponding
indexing structure to find matches.  The query processor converts data
stored in
XML to vectors, and computes various derived features from raw data
stored in the XML file, such as color and edge histograms and their
low dimension representations from images. It also generates vector
representations from text data.

\subsection{Indexing Structures}
The second group of components are indexing structures. They take
inputs from the query processor in the form of vectors, and produce
ordered lists of items. They sort items using measures between the
query template and objects stored in Robot Google and return the
list. 

Currently, eleven indexing structures exist in Robot Google. Six of
them are built for color histograms (each object contains two images,
and there are three color channels in an image); two of them are built
for edge
histograms; one of them is built for text data produced from an LSI
process; one of them is built for dimensional information of the
object, i.e., length, height, width, and the cube root of the product
of the three; the last indexing structure is built for the weight
measure of one object. Currently, $k$-$d$ trees have been used to
index all fields except the weight measure, where a binary
tree is used. In all $k$-$d$ trees, branch dimension is
selected in the round-robin fashion, starts from the left most
entry/first 
dimension in a vector. This is due to the fact
that eight histograms in one object are approximated by
polynomials in which high order terms contribute more to the
shape of the polynomial. Text data are processed by the LSI process,
where at its 
core is a singular value decomposition (SVD), which has the same
property that high order terms capture more information than low order
ones. All indexing components return fifteen items for each query,
except the text indexing and dimensional indexing component, in which
thirty items are returned. 

There are a few advantages for separating the
indexing structure into distinct subclasses.  First, this design
allows 
Robot Google to support incomplete queries, i.e., queries with missing
attribute fields, quite easily. For instance, if a query template
contains only one image instead of two, indexing structures built for
color and edge histograms for the missing image would not be used.  All
other Robot Google components would work the same as before.  Second,
it brings great flexibility to Robot Google as each indexing
structure can be added, removed or modified without changing other
components. New indexing structures can be added to support new data
types and existing indexing structures can be removed if they are
found to be not effective for retrieval. Different data types may
require different types of indexing structures. If an existing
structure is found to be less than ideal, it can be modified or
replaced. Third, it brings more parallelism into a query process
workflow. Each indexing structure processes one object attribute. Thus
there is no intervention between any two indexing structures. A
parallel process reduces search time for queries.

\subsection{Cross Analyzer}
The cross analyzer takes item lists from each indexing component, and
cross analyzes them to produce a single sorted list.  This list is
produced based on a weighted summation of distances between a query
template and items stored in the Robot Google database. 

The cross analyzer first creates a list contains all items generated
by indexing structures, without duplication. It then computes a
distance measure from the query template to every item in the
list.  The distance measure is a weighted $L_{1}$ norm, which can be
expressed as, the overall distance $D(A,B)$ between two objects $A$
and $B$ is equal to:
\begin{displaymath}
D(A,B) = \sum_{i=1}^{k}|w_id_i(A_i,B_i)|
\end{displaymath}
Where $k$ equals the number of item attributes presented in the query;
$w_i$ is the weight coefficient of the $i$th component; and $d_i$ is
the distance between $i$th components in the two objects.  All
$d_i(A_i,B_i)$ are computed using an $L_{1}$ distance measure, where
\begin{displaymath}
d_i(A_i,B_i) = \sum_{j=1}^{k}|A_{i,j} - B_{i,j}|
\end{displaymath}
For all image histograms represented by polynomial coefficients, k
equals 5; for the text data and the dimensional data fields, k
equals 4; and for the singleton weight measure, k equals 1. We
have evaluated a set of distance measures for this computation,
including $L_{2}$, $L_\infty$, Mahalanobis, and Kullback-Leibler
\cite{Castelli2002,puzicha99}.  $L_{1}$ gives the best compromise
between computation efficiency and accuracy.

The cross analyzer provides Robot Google a simple control on item
rankings through weight coefficients $w_i$. Since each indexing structure
processes one item attribute, if for some reason, a certain attribute
needs to be emphasized over others, the weight coefficient associated
with this attribute can be adjusted accordingly.  If we consider each
item as a point in some high dimensional space, it can be viewed that
weight coefficients can be used to dynamically enlarge or shrink the
space in different dimensions. For instance, if we want to ignore a
certain attribute in a query sample, we put a small weight coefficient
for this attribute. This is equivalent to shrinking the space in
dimensions used to represent this attribute.  When a space is shrunk,
the coordination of a point and the distance between two points no
longer matter as all points are crowded together in this space. 

We are at the very beginning stage to develop a systematic approach
for computing weight coefficients $w_i$. Currently, a static analysis
approach is taken. We design experiments for various data conditions
and query types. In each experiment we evaluate Robot Google
performance using the standard information retrieval measures:
precision and recall \cite{wiki:precision}.  We then search for weight
coefficients that maximize these measures.  The searching algorithm we
have implemented is an $n$-dimensional binary search, which is a good
compromise between simplicity and performance. 

The cross analyzer controls the number of items returned by Robot
Google.  Since the cross analyzer sorts the item list returned by
the indexing structures, two commonly used search types:
$k$-Nearest-Neighbor search, i.e., using distance measure $D$, find
$k$ objects that are closest to the query and within-Distance
($\alpha$-cut) search, i.e., using distance measure $D$, find all
objects which are within $\alpha$ to the query template, are
effortlessly supported.

\subsection{Response Formalizer}
The response formalizer is the last component in the Robot Google
query workflow.  It takes inputs, which are sorted item lists, from the
cross 
analyzer, transforms them into a format which can be easily understood
by robots, and then packages them into files. Snapshots of items,
which contains basic information such as dimensional measure, and URLs
which point to sources of items are both included. Containing
snapshots in the returned file help robots to filter out unwanted
results and find interesting information faster. 

\section{EXPERIMENT}
To evaluate the performance of Robot Google, the common measures of
precision and recall are used.  Precision in information
retrieval is defined as: 
\begin{displaymath}
precision = \frac{|{relevant document} \bigcap {retrieved
    documents}|}{|retrieved documents|}
\end{displaymath}
It is a measure of the percentage of results that are desired in the
total retrieved list. Recall in information retrieval is defined as:
\begin{displaymath}
recall = \frac{|{relevant document} \bigcap {retrieved 
    documents}|}{|relevant documents|}
\end{displaymath}
It is the percentage of desired retrieved results in the entire
database.

\subsection{Object Knowledge}
In this section, Robot Google performance on object data are
examined. Sixteen objects, including four bowls, one cup, two forks,
three knives, two plates, and four spoons are selected as data.  Two
images are taken of each object: a top view, and a side view.  All
images are taken against a white background and manually
segmented. Images are stored in JPEG format in the size of
640x480 pixels. For each image, histograms of its color
in RGB channels are computed, respectively, with 256 bins for each channel. The
Sobel edge detection algorithm is applied to compute the distribution
of edge orientations, and a histogram with 256 bins is obtained. To
simulate the uncertainty of the real world, twenty-nine duplications
of each object are created by adding 20\% random noise from a
Gaussian distribution to each object. The total sample size reaches
480. 

Six groups of performance evaluation are presented. In tests groups one
through three, relevant documents are defined as items which are
generated from the same master copy by adding Gaussian noise. Group
one shows the result where all documents returned from Robot Google
are considered.  Group two shows the result where only the top 30 most
relevant items are considered. Test three shows the result where
optimal weight coefficients are applied to rank returned items. Groups
four though six use same settings, except ``relevant documents'' are
defined as items which are in the same object class, e.g., fork, bowl,
etc. Each group contains four tests. Test one examines Robot
Google under optimal conditions, in which both query templates 
and objects stored in Robot Google contain complete information. Test
two tests performance with incomplete queries, in which one image is
missing in query templates. Test three tests performance with an
incomplete data record. One third of Robot Google records miss one
image in their record. Test four tests performance with both
incomplete queries and incomplete records. Query templates miss one
image and one third of Robot Google records miss the other image. Each
test is composed of 200 runs. In each run, one object is randomly
selected from the database as the query template. Average precision
and recall are computed.

\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt1.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 1. All items returned from Robot
  Google are taken into consideration. Items derived from the
  same instance are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test1}
\end{figure}
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt2.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 2. Top 30 items returned from Robot
  Google are taken into consideration. Items derived from the
  same instance are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test2}
\end{figure}
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt3.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 3. Top 30 items returned from Robot
  Google are taken into consideration. Optimal weight coefficients are
  applied for ranking. Items derived from the
  same instance are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test3}
\end{figure}
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt4.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 4. All items returned from Robot
  Google are taken into consideration. Items belong to the same
  object class are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test4}
\end{figure}
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt5.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 5. Top 30 items returned from Robot
  Google are taken into consideration. Items belong to the same
  object class are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test5}
\end{figure}
\begin{figure}[thpb]
  \centerline{
    \mbox{
      \psfig{file=fig/tt6.eps,width=3.8in}
    }
  }
  \caption{Performance Test Group 6. Top 30 items returned from Robot
  Google are taken into consideration. Optimal weight coefficients are
  applied for ranking. Items belong to the same
  object class are relevant. Tests range from complete query \&
  complete database to incomplete query \& incomplete database.}
  \label{Figure:test6}
\end{figure}

Many observations have been made during our performance evaluation. In
terms of precision, Robot Google shows good performance when the top
items from returned list are considered.  (Even though it is not shown
here, when the top 10 items are evaluated, precision reaches 100\%
uniformly for every test.) In general, performance drops when queries
are incomplete or database records are incomplete. However, we also
have discovered that certain features are more useful than others when
retrievals are concerned. For samples from our dataset, images
and text descriptions are better classifiers than dimensional
and weight measures. Hence missing images or text descriptions in a query shows
more negative impact to retrieval performance than missing dimensional
measures. The weight coefficient adjustment in the cross analyzer is a
very effective tool for improving retrieval performance. It identifies
attributes that have more classifying power, hence prevents or reduces
interference from other attributes. To achieve a uniformly high
performance on precision, a set of weight coefficients should be
pre-computed, based on the type of queries, i.e., attributes that are
missing in a query.

\subsection{Activity Knowledge}
In addition to object recognition, human activity recognition is
another field Robot Google can be used. Collaborating with
Prof. Dillmann's humanoid robot research group at the University of
Karlsruhe, we have 
obtained data generated by the {\it VooDoo human motion capture
system} \cite{Knoop2007, Knoop2006}, which gathers data of the human
configuration over time, resulting in 3D trajectories for every
modeled limb and joint angle of the human body.  These data contain 
8 activities with each activity performed multiple times,
resulting 120 instances of activities. Since all instances are
performed by a human experimenter, recorded lengths of instances range
from 41 frames to 151 frames. These 120 instances are stored into an
activity database.

Unlike the Feed Forward Neural Network (FFNN) classification model
used in \cite{Knoop2007}, Robot Google takes a simpler approach. In
{\it VooDoo}, the human body is represented by 19 4-$by$-4
transformation matrices, where each matrix describes the state of a
limb joint. In each transformation matrix, the upper left 3-$by$-3
sub-matrix describes the rotation of the joint, the right most column
describes the movement of the joint.  (See \cite{Knoop2006} for
a complete discussion of the {\it VooDoo} system representation.)  We
exploit two of these matrices: one that describes the trunk of the
body transformation and the other that describes the right forearm 
transformation, from each activity instance frame.
The motion description is based on six values from each of the two
transforms: 3 three diagonal elements of the rotation matrix and the
three translation components.  This results in 12 feature vectors.
We then approximate the
trajectory of every feature field across frames of an activity
instance by a fourth order polynomial and index every instance of a
trajectory into a $k$-$d$-tree.  Therefore, in contrast to the
eleven index structures developed for object knowledge
sharing, a twelve index structure approach is adopted for
activity recognition.


\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
{\bf Activity}&{\bf Correct}\\
\hline
\hline
Hold Out Hand & 91.0\% \\
Hold Out Object & 95.5\% \\
Put Object On Table & 89.9\% \\
Read Book & 73.6\% \\
Sitting & 89.9\% \\
Standing & 86.3\% \\
Take Object From Table & 77.7\% \\
Typing On Laptop & 100\% \\
\hline
\end{tabular}
\caption{Results for activity recognition experiments.}
\label{Table:act_exp}
\end{center}
\end{table}

We randomly select query templates from the activity database. Since
the purpose of activity recognition is to identify human activities,
Nearest-Neighbor search is more appropriate than
$k$-Nearest-Neighbor search or $\alpha$-cut search described in
previous sections. We then limit the number of returned activities for
each search to be 2 (since every search always returns the query
template itself as the first activity) and define the classification
as correct if the second returned item is the same activity as the
query template.  Results are presented in 
Table~\ref{Table:act_exp}. \cite{Knoop2007} indicates their FFNS
approach reaches average correctness ranges from 53.1\% to 100\% for
various activities; the Robot Google approach is comparable. 

\section{CONCLUSION}
This paper introduces the novel idea of developing robot virtual
machines and robot search engines for robot knowledge sharing and
discusses the architecture of the robot search engine we
have developed. As a proof of the concept system, it demonstrates the
merit of taking a sensor data grounded approach and using a flexible
architecture. Robot Google shows promising performance in our object
knowledge sharing experiments, where the search precision reaches
90+\% for the items selected as most relevant.  We have also examined
Robot Google's performance with robot activity knowledge, in which
Robot Google is used as an activity recognizer. In these experiments,
Robot Google demonstrates comparable results to activity recognizers
built by our colleagues at the University of Karlsruhe, in which a
neural network approach is used.

Future research includes:
\begin{itemize}

 \item
  development of virtual reference models for robots

  \item
  the study of Robot Google's performance on implemented, distributed
  robot systems

  \item
  investigation of a relevance feedback based approach for ranking
  returned items

  \item
  evaluation of the feasibility of porting this framework to
  intelligent software agent systems

  \item
  engagement of the robotics community into semantic robot web
  activity.
\end{itemize}

\section{ACKNOWLEDGMENTS}
We would like to thank Prof. R\"{u}diger Dillmann's group at the
University of Karlsruhe for sharing data with us, and in particular,
Steffen Knoop and Pedram Azad.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%\addtolength{\textheight}{-3cm}   % This command serves to balance the column lengths
                                  % on the last page of the document manually. It shortens
                                  % the textheight of the last page by a suitable amount.
                                  % This command does not take effect until the next page
                                  % so it should come on the page before the last. Make
                                  % sure that you do not shorten the textheight too much.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,bibliography/bib_file}

\end{document}
