Architecture of the regional datastore in Trinity College Dublin - Coghlan, B; Walsh, J; Childs, S; Quigley, G; O'Callaghan, D; Pierantoni, G; Ryan, J; Simon, N; Rochford, K (2009)
This paper proposes an architecture for the e-INIS regional datastore at TCD (Trinity College Dublin), one part of the larger federated national datastore activity of the e-INIS project, aimed at building a sustainable national e-Infrastructure for the Irish academic research community. e-INIS is funded under the Irish Higher Education Authority’s Programme for Research in Third-Level Institutions (PRTLI), a component of the National Development Plan. developed by the e-INIS project. The datastore builds upon existing infrastructure and services, including Grid-Ireland, the National Grid Initiative and EGEE, Europe’s leading grid infrastructure. It assumes users are in distinct research communities and that their data access patterns can be described via two properties, denoted as mutability and frequency-of-access. The architecture is for a back-end –individual academic communities are best qualified to define their own front-end services and user interfaces. The e-INIS regional datastore at TCD is run by the Operations Centre (OpsCentre) of Grid-Ireland [4], the Irish NGI (National Grid Initiative). It is actively involved in EGEE (Enabling Grids for E-SciencE) [3] and its planned successor EGI (European Grid Initiative) [1] and uses EGEE’s gLite middleware, which includes highly scalable gridenabled data management components. Grid-Ireland’s OpsCentre runs a set of national services, such as catalogues and resource brokers [4], as well as a grid site that hosts a 768-core cluster and >300TB storage. The Integrated Rule-Oriented Data System (iRODS) [2] was chosen to manage the national federated datastore. iRODS provides a global namespace for digital repositories and allows communities to define policies for their collections in the form of rules and state information. The iRODS catalogue service (iCAT) will be hosted in the OpsCentre and will federate resources provided by a number of iRODS servers residing in the regional datastore back-ends. The deployed grid technologies treat all data objects as immutable. Since this data will never be modified, only deleted, remote read caching and/or replication will provide very high performance, allowing use of bottom-of-therange JBOD (Just a Bunch of Disks) technology, far cheaper than storage for mutable data. Hence, a distinction is made between the immutable pattern typical of accesses to raw datasets and other mutable patterns (e.g. those typical of accesses to derived datasets). The distinction between access patterns will allow assignment of space reservations to the most cost-effective back-end technologies. This can be automated using iRODS rules. The other factor affecting storage strategy is how frequently storage is accessed and hence the class of storage required. Classes are denoted as online, nearline and offline and respectively are most appropriate for frequent (lowlatency) accesses, occasional accesses where some latency is acceptable and rare accesses such as retrieving data from deep archives. The initial offering of e-INIS will be online storage but in later tranches nearline storage will be investigated further, e.g. using MAID technology to power down idle disks. Offline storage is outside the scope of the pilot. To facilitate their front-ends there will be a separate bridge VM for each community which will be constrained to only access a subset of the data in the back-end but export it (through secure VPN) using whichever protocol they require. The intention is to ensure that the communities are insulated from the back-end and from each other in order to ensure quality of service and to decouple their front-end implementation from site-specific back-end implementations.