History

04/11/09 Portal Design Meeting

  • Attendees: Christoph, Jolyon, Max, Saeed, Thierry
  • Attachement: attachment:Portal-Brainstorming-091104.pdf
  • Goal: Brainstorm about the 1st part of Portal Design
  • 1st Part:
    • Portal to focus only on Experiment Monitoring / Feedback to user (Portal as an interface for Resource Discovery/Reservation will be addressed later)
    • EC is ran by user on his/her machine, do not assume that the machine is the console of the testbed (EC ran by Portal as "batch experiment execution" to be addressed later)
  • Design Directions:
    • Separation of concerns
    • Allow involved entities to be provided by separate institutions
    • Allow multiple instances of entities (i.e. instances are from different institutions)
  • EC is the entity responsible for experiment control, thus EC is the entity informing the Portal about Start/End of experiment
  • Scenario:
    • User start EC to run an experiment. User provide URI of experiment to EC
    • URI can refer to:
      • a local experiment description on the EC local disk (i.e. script) with a potential set of dependency script (e.g. stdlib.rb in our current OMF)
      • an experiment description which is hosted remotely on the portal
    • EC decides where to run the OML server for this experiment
      • Note: in federated environment, there may be many OML server organised hierarchically for a given experiment run. The OML server referred is the one on top. In a single environment as of now, this is the normal OML server running on the testbed
    • EC contacts Portal and sends to it:
      • the contact details of the OML server for this experiment
      • the Experiment ID
      • the URI for this experiment
      • if URI referred to local script(s) then upload it(them) to Portal
    • Portal:
      • if URI referred to remote scripts present on the Portal repository (no local scripts uploaded by EC)
        • Meaning: User wants to replay an existing experiment
        • Portal sends back to EC copies of the scripts corresponding to the URI
      • if URI referred to one or more scripts that have local user version and remote (Portal) version, which differ from each other
        • Meaning: User made modification to an existing experiment and want to run this modified one
        • Portal creates a new entry in its repository for this new version of this experiment and saves the associated script.
        • Important: we agreed here that each version modification is an independent entry!
        • Consequently: if user A & B of the same project make parallel modifications, each of their scripts will be saved as a new entity, thus it does not matter if their version conflict
        • Note: experiment repository does not need to assure that latest script version is indeed the one that people should run! Someone can decide that version 123 is the one for is paper result, even though someone else on the same project as created a completely different version 124.
      • if URI referred to local script(s) only (no remote version on the Portal
        • Meaning: this is a new experiment
        • Portal creates a new entry in its repository for this new experiment and its script(s)
    • EC gets some scripts back from the Portal (or only some OK signal if all local scripts should be used)
    • EC goes and execute the experiment
    • While experiment is running, EC contacts OML server and pushes into the experiment database:
      • Any experiment property states, and their changes
      • Any EC log messages
      • Other informations (e.g. start date, URIs of used scripts, etc...)
    • Portal may regularly pool the OML server to retrieve experiment database and give user an updated view of the experiment while it runs
    • At the end of the experiment, EC contacts Portal again and informs it that experiment is finished
    • Portal pool OML server to retrieve experiment database, present final info to user, and stores a copy of this database
    • Portal provides user with a way of "tagging" the experiment as "meaningful/success/good" or not. Status of the Tag of an experiment should be captured in the stored experiment database of the Portal.
  • Issues:
    • OML Server needs to be accessible by the portal. This is not really a problem, firewall port(s) can be opened for OML server(s) by hosting institutions
    • Implementation of Portal experiment script repository:
      • multiple discussions about using: GIT, custom version handling, Redmine internal repository
      • no consensus was achieved.
      • Max mentioned that maybe we should use some simple form of custom version handling
    • EC needs some modifications to perform the above scenario. A Feature Ticket was created on OMF issue tracker. This is not urgent for now, and is not to be added to release 5.2
    • NOTE: how to achieve repository synchronisation, for manual runs and for scheduled runs -> this is an issue related to resources discovery/reservation, and thus will not be addressed in this 1st part.
  • Follow up
    • Rodney to review affect on Portal specification and what has already been done
    • Max to review if we need to have a pause on InContext work ...
    • Saeed ?
    • Jolyon (with inputs from all) to advise on the solution to select for the Portal experiment script repository