cpubroker man page
SYNOPSIS
The CPU Broker is a reservation-based resource manager for CPU time.
The broker mediates between multiple real-time applications and the
facilities of an underlying real-time operating system: it implements a
configurable CPU management policy, monitors resource use, and adjusts
allocations over time. This architecture helps to ensure that the
quality of service (QoS) needs of "important" applications are satis-
fied, especially when the relative importance of applications changes
dynamically. An additional and important benefit is that the broker
can automatically determine many of the scheduling parameters of its
client applications.
This document provides an overview of the use and implementation of the
CPU Broker. The first section below is a "quick start" example of the
Broker in action. The second section summarizes the implementation of
the Broker, and introduces important terminology. The third section is
a brief tutorial on writing applications that interface with the CPU
Broker.
QUICK START
[Note: This demo requires most of the software dependencies described
in the README: TimeSys Linux, ACE/TAO, QuO, gkrellm v2, etc.]
If you would like to quickly see the Broker in action, you can run the
following commands in separate terminals to start the Broker, a syn-
thetic CORBA server, and its client:
[nemo@anemone /tmp] broker_allup_default
Done!
[nemo@anemone /tmp] rktimes -n rt1 -p 8080 -- rt_server -n rt1
CPU frequency: 851.947Mhz
RT server ready, waiting for connection from rt_client...
[nemo@anemone /tmp] rt_client
You can then monitor the CPU usage and reservation for rt_server using
gkrellm version two (available from http://www.gkrellm.net).
[nemo@anemone /tmp] gkrellm -s localhost -P 8080
The gkrellm window should display two "CPU" graphs: "CPU 0" is the CPU
usage of the rt_server program, and "CPU 1" is the reservation made by
the broker for that application. (Note that the labels are below the
graphs, not above.) Furthermore, the "CPU 1" graph shows the unused
portion of the reservation in green and overlays the used portion in
cyan. Both graphs should show square waves after a while, with the
reservation being slower to drop off. The important thing to note is
that the reservation changes over time to meet the needs of the appli-
cation. This is different from an ordinary reservation-based system
without brokering, in which the application's reservation would nor-
mally remain constant from the beginning to the end of execution.
Going through each step in detail, the first program, bro-
ker_allup_default, starts a process to house most of the broker objects
and connect the objects in a default configuration. Next we start a
program, rktimes, that monitors rt_server and sends updates to gkrellm
though TCP port 8080. The rt_server is a synthetic CORBA server that
will consume 250 or 500 milliseconds of CPU time every second and
reports this usage to the Broker. Finally, we start rt_client, which
connects to rt_server and periodically does an RPC that causes the
server to actually consume CPU time.
A primary feature of the Broker is its ability to manage multiple
applications and arbitrate between them when there is CPU contention.
Continuing with the previous (and running) example, the following com-
mands will demonstrate the Broker's ability to deal with contention.
First, we create some (artificial) contention by decreasing the total
amount of CPU the Broker will reserve for applications from 75% to 60%:
[nemo@anemone /tmp] cbhey file://strict_policy.ior set max-cpu to 0.60
The cbhey command is an extremely flexible tool that can communicate
with different parts of the Broker; we will be using it extensively in
later examples. Next, we add a new application by starting another
rt_server, rt_client, and gkrellm.
[nemo@anemone /tmp] rktimes -n rt2 -p 8081 -- rt_server -n rt2 -o
rtserver2.ior 200ms
[nemo@anemone /tmp] rt_client -s file://rtserver2.ior
[nemo@anemone /tmp] gkrellm -s localhost -P 8081
This new application consumes 200 milliseconds of CPU time every period
(1 second) and its reservation should also be around 200 milliseconds
per second. The contention created by this new application is
reflected in the gkrellm graph for "rt1" as it is receiving a lower
reservation than before. This occurs because both applications have
the same level of importance and the Broker cannot decide which one to
favor, so it arbitrarily chooses "rt2". You can change this by setting
the priority of one of the applications, referred to as "rt1" and
"rt2". For example, to bump the priority of "rt1" to one:
[nemo@anemone /tmp] cbhey file://strict_policy.ior set priority of task
rt1 to 1
The graph for "rt1" should return to normal and the graph for "rt2"
should show the Broker automatically dropping the reservation when the
reservation for "rt1" increases. You might try raising the priority of
"rt2" above "rt1" to see what happens.
IMPLEMENTATION
The CPU Broker is implemented as a CORBA-based server that receives
resource requests from soft real-time applications and which "trans-
lates" those into real reservations. An application that uses the Bro-
ker generates a request at the end of (some or all of) its task peri-
ods, when it knows of how much CPU it will need in future periods. An
application can be modified so that it communicates with the Broker;
alternatively, an unmodified application can be managed through the use
of a wrapper program, like proc_advocate(1), albeit at some loss in
precision. An application's request, along with any external data, is
fed into the Broker's adaptation strategies. Adaptation strategies for
individual applications pass their advice to a global strategy that
resolves contention for the CPU. Finally, when all of the adaptations
have been computed, the applications' CPU reservations are adjusted by
the Broker.
Communication between applications (or their wrappers) and the CPU Bro-
ker is done through remote method invocations on CORBA objects. At the
moment there are three main classes of Broker objects:
Manager
A Manager is a central object used to track other objects and
route messages between them. Typically there is only a single
manager per machine.
Task/Advocate
A Task object is the glue between the applications, the Broker,
and the real-time operating system's reservation scheme. Advo-
cates wrap Task objects and perform application-specific adapta-
tions.
Policy A Policy object makes global decisions about how to resolve con-
tention for the CPU. These objects take advice from the Advo-
cates and make the final decisions about how much CPU time every
managed process actually receives.
The standard implementations of these classes and their support code
are contained in several shared libraries:
libcpu-broker
Contains both a straightforward implementation of a Manager and
the base Task object, called RealTimeTaskImpl.
libtask_factories/libdelegates
Contains a variety of reusable Advocates.
libpolicies
Contains the StrictPolicy object, which implements a "strict
priority" contention policy that favors high-priority processes
when making reservations. For example, if all of the CPU has
been allocated and a high-priority process requests more time,
the StrictPolicy will reduce the reservations of lower priority
processes. In contrast, a low-priority process will have its
request lowered (and continually readjusted) to use whatever CPU
time is currently available.
The libraries contain the bulk of the CPU Broker's code and are used by
the tools listed below:
broker_allup
A "super server" that houses the shared libraries and the
objects that they create.
cbhey A "super client" that can be used to communicate with the vari-
ous Broker objects.
proc_advocate
A wrapper that allows unmodified applications to be managed by
the Broker.
rktimes
A generic utility for TimeSys Linux that monitors a resource set
and performs some of the jobs of rkexec(1).
Learning to wield these tools, especially cbhey(1), is key to using and
configuring the Broker to behave as desired in a variety of scenarios.
Consult the man pages for more information about these tools, and read
the Doxygen generated documentation in the install directory for more
details on the Broker's classes.
In addition to using the existing tools, effective use of the CPU Bro-
ker may require modifying applications to communicate with the Broker,
or writing new Broker objects that implement application-specific adap-
tation policies. The remainder of this man page is a brief tutorial
about writing applications that interface with the CPU Broker.
TUTORIAL
This tutorial gives you a detailed example of creating Broker objects,
creating your own Broker classes, and modifying an application. To
fully complete this tutorial, you will need the following software in
addition to the Broker: TimeSys Linux, ACE/TAO, QuO, OmniORB, python
v2, gkrellm v2, etc...
First, start the broker_allup_default process again, if it is not
already running. This executable is just a shell script that uses the
Broker's tools to construct a default configuration consisting of a
single Manager object that uses a StrictPolicy. Feel free to peruse
the script to see how it works.
The application we will be using is the rt_server, like in the QUICK
START section. However, this time we will be providing our own Task
object. Creating the object is simply a matter of asking the shared
library referenced by "cb.ior" and contained in the broker_allup
process:
[nemo@anemone /tmp] cbhey file://cb.ior create real-time-task with name
rt1 > rt1-rtt.ior
This command will construct an RealTimeTaskImpl object with the name
"rt1" and print its IOR on standard out. The name is used to easily
identify our application and will be the name of the RK resource set
that holds the application's processes. Next, we create an Advocate
object that does no adaptation, but gives us a place to insert other
advocates:
[nemo@anemone /tmp] cbhey file://delegates.ior create real-time-task-
delegate > rt1.ior
This new object acts as a delegate that passes all method calls on to
another object. Initially, it has nothing to delegate to, so you we
need to set the "remote-object" attribute to point at the Real-
TimeTaskImpl we just created.
[nemo@anemone /tmp] cbhey -c IDL:BrokerDelegates/Delegate:1.0
file://rt1.ior set remote-object to file://rt1-rtt.ior
This command line introduces some more cbhey syntax so lets take a
moment to examine the command in more detail. The purpose of cbhey is
to provide high-level access to the Broker's objects through a common
command line syntax. This approach makes it easier to avoid hard
wiring object configurations in applications since a user can construct
arbitrary graphs of objects just prior to execution time. These user
defined configurations can then be wrapped in a shell script for ease
of use, like broker_allup_default. The command line syntax is derived
from AppleScript's "tell" syntax, wherein a script communicates with a
particular "property" inside of a server. In our case, the server is a
CORBA object referenced by its IOR and the properties are the object's
methods and attributes. For example, in the line above we are setting
the "remote-object" attribute of "./rt1.ior" to an object reference.
The reverse is also possible, we can "get" the "remote-object"
attribute and have the IOR we just set printed to standard out. The
only departure from the English-like syntax is the "-c" option which is
used to perform a "cast" to another IDL type. This cast is required
because there are cases where cbhey is unable to figure out the
object's type and needs a hint from the user. See the man page for
more information and examples.
With our Advocate and Task object configuration now constructed we can
start our server, client, and gkrellm. Notice that we pass a different
option, "-t", to the rt_server. This option informs the server that
use an externally created Advocate instead of creating its own.
[nemo@anemone /tmp] rktimes -n rt1 -p 8080 -- rt_server -t
file://rt1.ior
[nemo@anemone /tmp] rt_client
[nemo@anemone /tmp] gkrellm -s localhost -P 8080
At this point, you may notice that the reservation being made by the
Broker is not the same as in the QUICK START section. The difference
is due to the fact that neither the RealTimeTaskImpl nor the advocate
are performing any adaptations. Instead, they are just passing the
value requested by the application. To get the same effect as before
we need to create a MaxDecayTaskAdvocate:
[nemo@anemone /tmp] cbhey file://task_factories.ior create max-decay-
task-advocate > rt1-mdta.ior
This advocate's strategy is to advise the Broker to set the reservation
to the highest value seen in the previous five periods. This policy
works well in general since it can quickly adapt to spikes in usage
without over-provisioning for long periods of time. Just like before,
we need to connect the object before using it:
[nemo@anemone /tmp] cbhey -c IDL:BrokerDelegates/Delegate:1.0
file://rt1-mdta.ior set remote-object to file://rt1-rtt.ior
We can then insert our new advocate and observe the change in the
reservation (there is no need to stop/start the application):
[nemo@anemone /tmp] cbhey -c IDL:BrokerDelegates/Delegate:1.0
file://rt1.ior set remote-object to file://rt1-mdta.ior
Creating your own delegate is a relatively simple process, you simply
overload a single method, adjust one of its parameters, and call the
same method on the wrapped object. You can make a simple advocate that
doubles the application's original request by saving the following
python code in the file "MyDelegate.py":
import sys
from omniORB import CORBA, PortableServer
from RealTimeTaskDelegateImpl import *
class MyDelegate(RealTimeTaskDelegateImpl):
def PassCPU(self, rtt, status, advice, krp):
# Call the wrapped object
advice.Compute = advice.Compute * 2
return self.myRemoteObject.PassCPU(rtt,
status,
advice,
krp)
pass
orb = CORBA.ORB_init(sys.argv, CORBA.ORB_ID)
poa = orb.resolve_initial_references("RootPOA")
di = MyDelegate()
do = di._this()
file = open(sys.argv[1], "w")
file.write(orb.object_to_string(do))
file.close()
poaManager = poa._get_the_POAManager()
poaManager.activate()
orb.run()
This program creates a subclass of a Broker-provided class, instanti-
ates an object of this subclass, and prints its IOR on standard out.
You can run it like so:
[nemo@anemone /tmp] /usr/local/bin/python MyDelegate.py rt1-md.ior
Again, we connect everything and watch the reservation behavior change:
[nemo@anemone /tmp] cbhey -c IDL:BrokerDelegates/Delegate:1.0
file://rt1-md.ior set remote-object to file://rt1-rtt.ior
[nemo@anemone /tmp] cbhey -c IDL:BrokerDelegates/Delegate:1.0
file://rt1.ior set remote-object to file://rt1-md.ior
At this point we will create our own application that is managed by the
Broker, so stop the rt_server and rt_client we started above. This new
application will also be synthetic, like rt_server, but it should give
you a good idea of what is required of an application that needs to
communicate with the Broker. Read and copy the following python code
into the file "brapp.py":
import os
import sys
from resource import *
from signal import *
from omniORB import CORBA, PortableServer
import Broker
orb = CORBA.ORB_init(sys.argv, CORBA.ORB_ID)
obj = orb.string_to_object(sys.argv[1])
# Get a reference to the Manager.
man = obj._narrow(Broker.Manager)
obj = orb.string_to_object(sys.argv[2])
# Get a reference to the Advocate we should use.
adv = obj._narrow(Broker.RealTimeTask)
# Get objects required to make CPU reports.
cr = Broker.CPUReserve(0, 0)
krp = []
def noop(sig, fr):
return
signal(SIGALRM, noop)
# Get our current resource usage for comparison
# later on.
last_usage = getrusage(RUSAGE_SELF)
# Setup our scheduling parameters
sp = [
# Period of one second.
Broker.NamedValue("period",
CORBA.Any(CORBA.TC_string,
"1s")),
# The process ID to manage.
Broker.NamedValue("pid",
CORBA.Any(CORBA.TC_long,
os.getpid()))]
# Add ourselves to the manager.
man.AddTask(adv, sp)
try:
alarm(1.0)
while 1:
pause() # Wait for a signal
alarm(1.0) # Setup a periodic signal
# Do something...
print "Hello, World!"
for x in range(0, 1000):
for y in range(0, 50):
pass
pass
# Get our current usage,
usage = getrusage(RUSAGE_SELF)
# ... subtract our previous usage, and
total = ((usage[0] + usage[1]) -
(last_usage[0] + last_usage[1]))
# ... report it to the broker (in
# microseconds).
cr.Period = 1000000
cr.Compute = int(total * 1000000)
adv.ReportCPU(cr, cr, krp)
last_usage = usage
pass
pass
finally:
# Always be sure to remove ourselves
# from the manager.
man.RemoveTask(adv)
pass
Start your new application by running:
[nemo@anemone /tmp] /usr/local/bin/python brapp.py `cat manager.ior`
`cat rt1.ior`
If everything worked you should see "Hello, World!" being printed out
every second.
Every application that uses the Broker follows the same basic steps:
(1) construct the scheduling parameters, (2) add the Advocate to the
Manager using AddTask(), (3) periodically call ReportCPU() on the advo-
cate, and (4) remove the Advocate from the Manager with RemoveTask()
when finished. The first step, constructing the scheduling parameters,
depends on the timing requirements of your application. For example, a
video viewer might have a period of 1/30th of a second. Applications
that do not have a natural period, like servers, should be set to the
smallest period of its clients, or some value that results in accept-
able timing behavior. The process ID that is specified need only be
one that is in the resource set to be managed. Thus, if you have a
multi-threaded application, you can ensure the other threads are added
to the set by creating them after the advocate is added. If this is
not possible, start the application already in a resource set using
rkexec(1) or rktimes(1). The broker will then use the existing
resource set instead of creating one. The last part of initialization
is passing the advocate and the scheduling parameters to the manager
using AddTask().
Once the application is added to the Broker, it can begin its periodic
processing and call ReportCPU(). In the above example, we figure out
the CPU usage of our process using getrusage(2) and then send that
value, in microseconds, to the advocate. Notice that this only gets
the resource usage for the current thread and not the others in the
process. To do this properly, we would have to call a Broker provided
utility function, rk_resource_set_get_usage(3), that computes the usage
for all of the processes/threads in a resource set. Unfortunately,
there is no python binding for this C function, so we have to compro-
mise and use getrusage.
Finally, when the periodic processing of an application terminates we
have to make sure to remove our advocate from the manager so other
tasks can use the newly freed resources. In case something goes cata-
strophically wrong and the application cannot perform the cleanup, the
Broker should be able to repair itself, at least partially. Any other
inconsistencies left by such an occurrence can be repaired manually
using cbhey and rkdestroyRS.
This concludes the tutorial, to learn more about the commands and func-
tions used, see the man pages listed below.
SEE ALSO
broker_allup(1), broker_allup_default(1), cbhey(1), proc_advocate(1),
rkexec(1), rktimes(1), rt_server(1), rt_client(1),
rk_resource_set_get_by_name(3), rk_resource_set_get_usage(3),
gkrellm(1)
AUTHOR
The Alchemy project at the University of Utah.
NOTES
The Alchemy project can be found on the web at
Man(1) output converted with
man2html