vo-centric ganglia catalin lucian dumitrescu, mike...

33
VO-centric Ganglia Catalin Lucian Dumitrescu, Mike Wilde*, Rob Gardner The University of Chicago * Argonne National Laboratory July - August 2k2

Upload: vudat

Post on 12-Sep-2018

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

VO

-cen

tric

Gan

glia

Cat

alin

Lu

cian

Du

mit

resc

u, M

ike

Wil

de*

, Ro

b G

ard

ner

Th

e U

niv

ersi

ty o

f C

hic

ago

* A

rgon

ne

Na

tion

al

La

bor

ato

ryJu

ly -

Au

gust

2k2

Page 2: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Wo

rk G

oal

sG

lob

al R

eso

urc

e U

sage

Vie

wer

(an

d n

ot

on

ly)

at t

he

Gri

d L

evel

Gri

d a

nd

VO

leve

l vie

ws

of

reso

urc

e ac

tivi

tyu

tili

zati

on

per

form

ance

pol

icy

poi

nt

of v

iew

: ach

ieve

d v

ersu

s p

rovi

ded

Ter

ms

Gri

d: c

oll

ecti

on

of r

eso

urc

e fr

acti

on

s (s

et o

f sit

es p

rovi

ded

for

usa

ge)

Exa

mp

le: A

TL

AS

grid

, CM

S gr

idV

O: c

oll

ecti

on

of i

den

tifi

able

en

titi

es e

nti

tled

to

use

gri

d r

eso

urc

esE

xam

ple

: AT

LA

S V

O (

use

rs, g

rou

ps)

, CM

S V

O (

use

rs, g

rou

ps)

Page 3: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Req

uir

emen

tsE

ven

t lo

ggin

gs a

nd

sta

tist

ic v

iew

s fo

rm

ach

ines

, clu

ster

s, s

ites

, gri

ds

use

rs, g

rou

ps,

VO

s, w

ork

typ

esjo

bs

Op

erat

ion

s(t

o a

chie

ve s

cala

bil

ity

in p

rovi

din

g m

ean

ingf

ul i

nfo

rma

tion

)re

trie

ve /

co

llec

tag

greg

ate

sto

revi

zual

ize

en

d u

ser

/ in

telli

ge

nt

job

sub

mis

sio

n p

oin

t /

po

licy

en

forc

em

en

t

Page 4: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Pre

sen

tati

on

Ou

tlin

e1.

In

tro

du

ctio

n, G

oal

s &

Req

uir

emen

ts

2. E

xist

ing

Wo

rk O

verv

iew

3. A

cco

mp

lish

ed W

ork

a. D

ata

Su

mm

ati

on a

nd

Hie

rarc

hic

Rep

orti

ng

b. M

DS

Ad

dre

ss P

ub

lish

ing

c. L

oca

l Sen

sors

Sp

ecif

ica

tion

d. D

ata

Ret

riev

al f

rom

Na

gios

4. S

cree

nsh

ots

/ D

emo

in a

Sim

ula

ted

En

viro

nm

ent

5. N

agio

s W

eb I

nte

rfac

e In

corp

ora

tio

n

6. C

urr

ent

Stat

us

& C

on

clu

sio

ns

& R

emai

nin

g &

Fu

ture

Wo

rk

Page 5: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Gan

glia

Ove

rvie

wR

eal-

tim

e m

on

ito

rin

g (a

nd

exe

cuti

on

en

viro

nm

ent)

run

s o

n a

sin

gle

ho

stco

llec

ts d

ata

fro

m o

ther

ho

st-l

evel

mo

nit

ori

ng

too

ls (

e.g.

, gm

on

d, r

ock

s,n

agi

os-e

xpor

ter)

or

even

oth

er g

angl

ia d

aem

on

sd

ata

is m

anip

ula

ted

in X

ML

fo

rmat

and

sto

red

loca

lly

in a

RR

D fo

r st

atis

tics

pu

rpo

ses

use

s h

ost

-lev

el c

oll

ecto

rs (

gmo

nd

s) f

or

loa

d,

me

mo

ry,

swa

p,

dis

k,n

etw

ork

, u

ser

de

fin

ed

me

tric

s

com

pu

te n

od

es

com

pu

te n

od

es

RR

D

qu

erie

s

gan

glia

da

emon

gmon

d d

aem

ons

gmon

d d

aem

ons

Page 6: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Web

In

terf

ace

Ove

rvie

wP

rovi

des

an

eas

y-to

-use

an

d p

ow

erfu

l mec

han

ism

to

vie

wgr

aph

ical

ly g

ath

ered

info

rmat

ion

Op

erat

ion

s:lo

cal d

ata

retr

ieva

lm

eta-

clu

ster

su

mm

atio

ns

dif

fere

nt

view

s (m

eta-

clu

ster

vie

ws,

clu

ster

ove

rvie

ws,

ho

st d

etai

ls)

Lim

itat

ion

s:o

nly

tw

o le

vel:

clu

ster

, ho

std

ata

mu

st r

esid

e o

n t

he

web

serv

er n

od

e

Mo

re in

form

atio

n:

htt

p:/

/gan

glia

.so

urc

efo

rge.

net

/

Page 7: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Nag

ios(

TM

) as

Alt

ern

ativ

eA

few

wo

rds

abo

ut:

Cen

tral

ized

mo

nit

ori

ng

too

l fo

r se

rvic

e st

atu

s d

ata

gat

her

ing

Co

mp

lex

spec

ific

atio

n f

iles

(ca

n b

e au

tom

atic

ally

gen

erat

ed)

Po

wer

ful w

eb in

terf

ace,

wic

h p

rovi

des

su

pp

ort

fo

r re

mo

teco

mm

and

s

Page 8: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Pre

sen

tati

on

Ou

tlin

e1.

In

tro

du

ctio

n, G

oal

s &

Req

uir

emen

ts

2. E

xist

ing

Wo

rk O

verv

iew

3. A

cco

mp

lish

ed W

ork

a. D

ata

Su

mm

ati

on a

nd

Hie

rarc

hic

Rep

orti

ng

b. M

DS

Ad

dre

ss P

ub

lish

ing

c. L

oca

l Sen

sors

Sp

ecif

ica

tion

d. D

ata

Ret

riev

al f

rom

Na

gios

4. S

cree

nsh

ots

/ D

emo

in a

Sim

ula

ted

En

viro

nm

ent

5. N

agio

s W

eb I

nte

rfac

e In

corp

ora

tio

n

6. C

urr

ent

Stat

us

& C

on

clu

sio

ns

& R

emai

nin

g &

Fu

ture

Wo

rk

Page 9: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Hie

rarc

hic

Rep

ort

ing

Th

e B

ig P

ictu

re:

gm

on

dgm

on

dgm

on

d

gan

glia

RR

D

gmo

nd

gmo

nd

gmo

nd

gan

glia

RR

D

gmo

nd

gmo

nd

gmo

nd

gan

glia

RR

D

gan

glia ga

ngl

ia

gan

glia

...

EU

gri

d

US

grid

Page 10: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Dat

a Su

mm

atio

nP

rob

lem

:gr

ids

are

larg

e co

llec

tio

ns

of c

om

pu

ters

mo

nit

ori

ng

info

rmat

ion

is d

iffi

cult

to

be

pas

sed

aro

un

dst

and

ard

gan

glia

do

es n

ot

pro

vid

e su

pp

ort

fo

r la

yere

d r

epo

rtin

g

Ap

pro

ach

ed s

olu

tio

n:

gath

er in

form

atio

n, s

um

an

d p

rovi

de

on

ly a

n o

verv

iew

Ho

w (

seve

ral t

ypes

of

dat

a):

stri

ngs

nu

mer

ics

sum

(m

emo

ry, c

pu

nu

mb

ers,

..)

aver

age

(cp

u_l

oad

, ...)

Page 11: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Lo

cal S

enso

rsA

s a

mea

ns

for:

cap

turi

ng

add

itio

nal

info

rmat

ion

:ga

teke

eper

s /

clu

ster

s /

site

s /

grid

s sp

ecif

ic d

ata,

e.g

.:ga

teke

eper

ad

dre

sses

an

d t

ypes

po

licy

info

rmat

ion

use

rs, g

rou

ps,

an

d V

Os

usa

ge s

tati

stic

s

min

imal

inte

ract

ion

wit

h o

ther

loca

l so

ftw

are

com

po

nen

ts

dis

tin

ctio

n o

f in

form

atio

n p

ub

lish

ed t

hro

ugh

XM

L

In a

nu

tsh

ell,

no

t a

req

uir

emen

t, b

ut

an u

sefu

l fea

ture

Page 12: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

MD

S A

dd

ress

/Ret

riev

alM

oti

vati

on

:h

ard

to

mai

nta

in c

entr

aliz

ed c

on

figu

rati

on

fil

es (

Nag

ios)

imp

oss

ible

to

cap

ture

dyn

amic

ch

ange

s o

f th

e en

viro

nm

ent

real

co

nn

ecti

on

wit

h t

he

Gri

d c

om

po

nen

ts

Tw

o o

per

atio

ns

wer

e in

tro

du

ced

:G

angl

ia D

aem

on

Ad

dre

ss P

ub

lish

ing:

by

mea

ns

of

a gr

id p

ub

lish

erG

angl

ia A

dd

ress

es R

etri

eval

: by

mea

ns

of

a sp

ecia

l mo

du

le t

hat

qu

erie

sG

RIS

/GII

S se

rver

s

Page 13: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Inte

rfac

ing

wit

h N

agio

sP

rob

lem

:d

iffe

ren

t si

tes/

grid

hav

e sp

ent

con

sid

erab

le a

mo

un

t o

f w

ork

tim

e in

imp

rovi

ng

oth

er m

on

ito

rin

g to

ols

(e.

g., N

agio

s) f

or

thei

r p

arti

cula

r n

eed

s

Fo

cus

on

Nag

ios:

info

rmat

ion

co

llec

ted

by

Nag

ios

is s

till

ava

ilab

le t

o V

O-c

entr

ic G

angl

iaid

ea: s

imu

late

a g

mo

nd

co

llec

tor

wit

h a

sim

ple

dae

mo

n p

roce

ss

Page 14: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Imp

rove

d W

eb I

nte

rfac

eSt

and

ard

inte

rfac

e m

od

ifie

d t

o h

and

le n

ew t

ypes

A s

eco

nd

vie

w t

ho

rugh

a s

imp

lifi

ed N

agio

s in

terf

ace

ach

ieve

d b

y ex

po

rtin

g in

tern

al d

ata

in N

agio

s fo

rmat

Page 15: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Pre

sen

tati

on

Ou

tlin

e1.

In

tro

du

ctio

n, G

oal

s &

Req

uir

emen

ts

2. E

xist

ing

Wo

rk O

verv

iew

3. A

cco

mp

lish

ed W

ork

a. D

ata

Su

mm

ati

on a

nd

Hie

rarc

hic

Rep

orti

ng

b. M

DS

Ad

dre

ss P

ub

lish

ing

c. L

oca

l Sen

sors

Sp

ecif

ica

tion

d. D

ata

Ret

riev

al f

rom

Na

gios

4. S

cree

nsh

ots

/ D

emo

in a

Sim

ula

ted

En

viro

nm

ent

5. N

agio

s W

eb I

nte

rfac

e In

corp

ora

tio

n

6. C

urr

ent

Stat

us

& C

on

clu

sio

ns

& R

emai

nin

g &

Fu

ture

Wo

rk

Page 16: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Use

d C

on

figu

rati

on

blu

e.c

olo

rsre

d.c

olo

rsm

agen

.co

lors

yell

ow

.co

lors

pu

rple

.co

lors

Fer

mi C

lust

er @

Uo

fCP

hys

Sci S

ite

@U

ofC

Clu

ster

@A

NL

Gri

d @

Ch

icag

o

Site

@U

ofC

gmo

nd

gmo

nd

gmo

nd

gmet

adgm

etad

gmm

etad

2gm

met

ad2

gmm

etad

2

Page 17: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Mai

n P

age

Page 18: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Mai

n P

age

(2)

Page 19: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Ove

rvie

w

Page 20: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Det

ails

Page 21: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Det

ails

(m

em)

Page 22: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Sen

sors

In

fo

Page 23: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Pre

sen

tati

on

Ou

tlin

e1.

In

tro

du

ctio

n, G

oal

s &

Req

uir

emen

ts

2. E

xist

ing

Wo

rk O

verv

iew

3. A

cco

mp

lish

ed W

ork

a. D

ata

Su

mm

ati

on a

nd

Hie

rarc

hic

Rep

orti

ng

b. M

DS

Ad

dre

ss P

ub

lish

ing

c. L

oca

l Sen

sors

Sp

ecif

ica

tion

d. D

ata

Ret

riev

al f

rom

Na

gios

4. S

cree

nsh

ots

/ D

emo

in a

Sim

ula

ted

En

viro

nm

ent

5. N

agio

s W

eb I

nte

rfac

e In

corp

ora

tio

n

6. C

urr

ent

Stat

us

& C

on

clu

sio

ns

& R

emai

nin

g &

Fu

ture

Wo

rk

Page 24: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Ho

st D

etai

ls

Page 25: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Ser

vice

s D

etai

ls

Page 26: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Scre

ensh

ot:

Sta

tus

Map

Page 27: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Use

d C

on

figu

rati

on

(2)

blu

e.co

lors

red

.co

lors

mag

en.c

olo

rsye

llo

w.c

olo

rsp

urp

le.c

olo

rs

Fer

mi C

lust

er @

Uo

fCP

hys

Sci S

ite

@U

ofC

Clu

ster

@A

NL

Gri

d @

Ch

icag

o

Site

@U

ofC

nag

ios

gmo

nd

gmo

nd

gmet

adgm

etad

exp

ort

ergm

met

ad2

gmm

etad

2

Page 28: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Inte

rfac

e w

ith

Nag

ios

Page 29: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Pre

sen

tati

on

Ou

tlin

e1.

In

tro

du

ctio

n, G

oal

s &

Req

uir

emen

ts

2. E

xist

ing

Wo

rk O

verv

iew

3. A

cco

mp

lish

ed W

ork

a. D

ata

Su

mm

ati

on a

nd

Hie

rarc

hic

Rep

orti

ng

b. M

DS

Ad

dre

ss P

ub

lish

ing

c. L

oca

l Sen

sors

Sp

ecif

ica

tion

d. D

ata

Ret

riev

al f

rom

Na

gios

4. S

cree

nsh

ots

/ D

emo

in a

Sim

ula

ted

En

viro

nm

ent

5. N

agio

s W

eb I

nte

rfac

e In

corp

ora

tio

n

6. C

urr

ent

Stat

us

& C

on

clu

sio

ns

& R

emai

nin

g &

Fu

ture

Wo

rk

Page 30: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Cu

rren

t St

atu

sE

very

thin

g is

dev

elo

ped

an

d t

este

d o

n m

y si

mu

late

d c

lust

er

Th

e so

ftw

are

is d

istr

ibu

ted

as

two

pac

kag

es a

vail

able

at:

htt

p:/

/peo

ple

.cs.

uch

icag

o.e

du

/~cl

du

mit

r/so

ft/m

on

i/v2

Pac

man

per

son

al c

ach

e at

:h

ttp

://p

eop

le.c

s.u

chic

ago

.ed

u/~

cld

um

itr/

pac

man

Page 31: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Inst

alat

ion

Ste

ps

Co

re (

5 st

eps)

:au

toco

nf

./co

nfi

gure

--p

refi

x=D

IRm

ake

mak

e in

stal

led

it b

y h

and

co

nfi

gura

tio

n f

iles

(in

stru

ctio

ns

insi

de

pac

kage

)

Intr

efac

e (i

ncl

ud

es b

oth

ext

end

ed a

nd

nag

ios

lik

e vi

ews

- 4

step

s):

auto

con

f./

con

figu

re -

-pre

fix=

DIR

2 --

wit

h-g

met

ad=

DIR

mak

em

ake

inst

all

Page 32: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Co

ncl

usi

on

s

Page 33: VO-centric Ganglia Catalin Lucian Dumitrescu, Mike …avery/griphyn/meetings/sc2002/ganglia_08302k2...Presentation Outline 1. Introduction, Goals & Requirements 2. ... Data Retrieval

Fu

ture

Wo

rkW

rite

oth

er s

enso

rs a

bo

ut

VO

res

ou

rce

usa

ges

Stat

isti

cs c

lass

ific

atio

n b

y:re

sou

rce

con

sum

pti

on

VO

s, g

rou

ps,

use

rsW

ork

Typ

ere

sou

rces

:h

ost

, clu

ster

, sit

e, a

nd

gri

d

Inte

grat

e w

ith

th

e ru

le-b

ased

po

licy

mo

du

le