IS Release v0.1 Anemone

Table of contents

IS Release v0.1 Anemone
2 May 2022. Release notes RRAP M&DS IS v0.1 “Anemone”
Detailed Release Notes

2 May 2022. Release notes RRAP M&DS IS v0.1 “Anemone”

Anemone

RRAP M&DS Information System (IS) v0.1 “Anemone” release includes system components that allows users to login, find out information about the M&DS IS, and interface with the RRAP M&DS Data Store.

Single sign on

The RRAP M&DS IS supports Single-sign-on using the Australian Access Federation (AAF), Google accounts, and Github identity providers. If you have a user account in one of those 3 identity providers, you will be able to access the RRAP M&DS IS and request authorisation to access RRAP M&DS IS tools, such as the Data Store.

Data Store v1.0

The Data Store facilitates registration and creation of datasets for RRAP M&DS. Uploaded data is stored in an AWS S3 bucket, which serves as a persistent storage repository.

Data Store v1.0

What can users do in the Data Store?

Users can now login to the RRAP M&DS Data Store to do the following.

Register datasets: Create a dataset record for a dataset relevant to M&DS. Dataset records could either point to a dataset that is managed externally (e.g. institutional repositories) or a location in the M&DS Data Store. Valid examples of dataset records include:

CSV files
a set of related NetCDF files

Upload files: Authorised users are able to upload files to a location registered in the Data Store. When a user registers a dataset, a digital identifier is minted for that dataset and a folder is created in the shared AWS S3 bucket. Users can then upload files to that location via AWS S3 tools. Learn more about RRAP M&DS IS Digital Identifiers here.

Discover datasets: Authorised users can access the Data store to search for datasets and view their details.

Download and synchronise datasets: Authorised users are able to access files on the Data Store and download them to a local computing environment using AWS tools and the AWS web console. Users can also synchronise dataset folders to a local computing environment, which only downloads updated files optimising file transfer. This will allow users to share data and synchronise data updates.

Automated archival: Datasets that are registered and uploaded to the Data store will be archived using automation tools. Archival will allow RRAP to manage these datasets for future reference.

Browse to the IS Landing Page: https://mds.gbrrestoration.org/

User documentation: https://gbrrestoration.github.io/rrap-mds-knowledge-hub/information-system/data-store/

Data record metadata schema

The Data Store v1.0 will have a minimal dataset metadata record schema using RO-CRATE as the metadata data file format. The minimal dataset metadata record fields include: name, description, publisher, organisation, licence, keywords. The metadata fields included in Data Store v1.0 enables data registration, upload, sharing via the S3 APIs to support current modelling activities in M&DS. This is shown in the diagram below. Future releases will incrementally add additional capability to capture RRAP M&DS, ISO and Science related metadata for when we need wider data publication.

RRAP Data Store Metadata schema v1.0

Contributors

Andrew Freebairn
Fareed Mirza
Jonathan Yu
Linda Thomas
Peter Baker
Peter Fitch
Ross Petridis
Sharon Tickell
Simon Cox

Citation

CSIRO (2022), RRAP M&DS Information System Release notes version 0.1 “Anemone”,

Detailed Release Notes

Feature

[RRAPIS-276] - Add copy button to AWS command and export fields in data store UI
[RRAPIS-295] - S.DS.1 User Login and Profile Fields
[RRAPIS-296] - S.DS.2.a Metadata record properties
[RRAPIS-297] - S.DS.2 Create dataset workflow
[RRAPIS-298] - S.DS.3 upload and reposit workflow
[RRAPIS-302] - S.DS.7 View a dataset
[RRAPIS-303] - S.DS.8 List Datasets
[RRAPIS-304] - S.DS.9 Resolve dataset identifier
[RRAPIS-305] - S.DS.10 Download dataset files
[RRAPIS-325] - Add source URL where the request came from in email for updated access roles
[RRAPIS-341] - Linux/windows credential copy format
[RRAPIS-370] - S.DS.1a Minimal User Login and Profile Fields

Story

[RRAPIS-39] - As a modeller, I want to register a dataset in the RRAP IS system so that it can be used in counterfactual modelling workflows
[RRAPIS-41] - As a RRAP IS build team, we want to understand who to talk to and questions to ask in the RRAP M&DS modelling and DS teams, so that we can understand data flows
[RRAPIS-45] - As a RRAP IS build team, we want to understand data flows from the other RRAP M&DS teams, so that we can understand data flows
[RRAPIS-157] - ID.1. As the data store application, I need to request a persistent identifier (e.g. a handle and other) to uniquely identify a dataset so that users and other applications are able to refer to the dataset using the persistent identifier.
[RRAPIS-158] - ID.2. As the data store application, I need to update a previously minted persistent identifier (handle) so that users and other applications are able to continue to refer to the dataset using the persistent identifier.
[RRAPIS-171] - How do users login and upload/register datasets in the RRAP IS?
[RRAPIS-172] - How does the IS system register the dataset?
[RRAPIS-174] - How do we describe datasets in some useful way?
[RRAPIS-255] - Create user documentation structure for RRAP M&DS IS in GH site
[RRAPIS-256] - Usability testing: As a user, I want to login to the RRAP Landing portal and request authorisation for access to data store
[RRAPIS-289] - US.DS.1 Registering datasets
[RRAPIS-290] - US.DS.2 Repositing datasets
[RRAPIS-291] - US.DS.3 Managing datasets
[RRAPIS-292] - US.DS.4. Find and view a dataset record
[RRAPIS-293] - US.DS.5.a Access dataset record files
[RRAPIS-294] - US.DS.5.b API Access to dataset record files
[RRAPIS-326] - As a user, I want to add register data that is managed outside of RRAP, so I can reference it and not register again
[RRAPIS-377] - US.RRAP.1. As a RRAP M&DS modeller, I would like to register datasets so that other modellers are able to use my outputs
[RRAPIS-378] - US.RRAP.2. As a RRAP M&DS modeller, I would like to access datasets that are produced by other modellers upstream from my work
[RRAPIS-385] - US.NFR.1.4. log in with existing institutional identity
[RRAPIS-386] - US.NFR.1.5. Single sign-on and shared sessions
[RRAPIS-391] - US.NFR.2.2. Access to user guides
[RRAPIS-404] - US.NFR.5.1. Consistent RRAP Visual Identity
[RRAPIS-405] - US.NFR.5.2. Support for Multi-Factor Authentication (MFA)
[RRAPIS-411] - US.NFR.6.4. Automated system deployment

Question

[RRAPIS-164] - Produce licence documents for RRAP IS related system components

Bug

[RRAPIS-244] - Handle service/data api response is misleading due to lack of auth
[RRAPIS-285] - Data store endpoints
[RRAPIS-286] - Data store UI token box
[RRAPIS-287] - Cross links from landing page are incorrect for non prod deployments
[RRAPIS-331] - Datastore logout doesn't log user out
[RRAPIS-334] - Landing page role access request doesn't respect failure on the response.
[RRAPIS-336] - On the profile/roles RRAP M&DS Landing portal links to prod from dev
[RRAPIS-339] - http website loads for static UIs
[RRAPIS-343] - The link to list datasets page in the breadcrumb in /dataset/item UI seems to be broken
[RRAPIS-464] - Login is lost on refresh of dev landing portal and data store

Epic

[RRAPIS-4] - IS-2.1 Data Store and Workflow Requirements
[RRAPIS-5] - IS-2.1 Provenance Requirements
[RRAPIS-6] - IS-2.1 Decision register requirements
[RRAPIS-9] - IS-2.1 Review previous work
[RRAPIS-38] - IS-3.x Define RRAP IS MVP
[RRAPIS-67] - IS-2.1 specification and design document
[RRAPIS-68] - IS-2.2 M&DS IS Cloud-hosting arrangements report
[RRAPIS-73] - IS-3.1 Data store v1.0 implementation
[RRAPIS-166] - IS-3.x Landing portal MVP v1.0 implementation
[RRAPIS-177] - IS-3.1 RRAP IS MVP Authorisation v1.0
[RRAPIS-250] - IS-3.1 Provenance store v1.0 Implementation
[RRAPIS-253] - IS-3.2 User documentation for v0.1 Release

Task

[RRAPIS-22] - Literature survey for provenance systems
[RRAPIS-30] - Attend and gather requirements from RRAP IS counterfactual modelling workshop
[RRAPIS-31] - Attend and contribute to cross-RRAP project workshop on coral metadata
[RRAPIS-32] - Attend and contribute to cross-RRAP data repository discussion
[RRAPIS-37] - Gather initial set of requirements for Decisions
[RRAPIS-54] - Engage and report back from RECOM/eReefs modelling discussion to understand some requirements for RRAP
[RRAPIS-55] - Investigate easy-to-use user uploads/sync'ing to AWS S3
[RRAPIS-59] - Organise a demo of RECOM to IS team
[RRAPIS-60] - Organise a demo of CoCoNet to IS team
[RRAPIS-62] - Facilitate discussions with CoCoNet team to understand input/output data flows
[RRAPIS-63] - Follow up discussion with RECOM/eReefs team to understand input/output data flows
[RRAPIS-64] - Facilitate discussion with ADRIA team to understand input/output data flows
[RRAPIS-65] - Facilitate discussion with ReefMod team to understand input/output data flows
[RRAPIS-66] - Organise a planning session on the RRAP M&DS IS Design Specification and System Design deliverable
[RRAPIS-69] - Document starting assumptions for data storage and usage requirement for M&DS modelling teams
[RRAPIS-70] - Organise a discussion about data store requirements and potential implementation options
[RRAPIS-72] - Initial provenance store user story mapping session
[RRAPIS-75] - Draft iteration of Decision register into spec document
[RRAPIS-76] - Draft iteration of Provenance store into spec document
[RRAPIS-77] - Create conceptual model of provenance records
[RRAPIS-78] - Collate specification and implementation into specification table(s)
[RRAPIS-79] - Investigate Balsamiq license
[RRAPIS-82] - Consolidate the provenance and data store user story maps
[RRAPIS-83] - Explore implementation options for provenance system
[RRAPIS-87] - Organise a check-in with RRAP stakeholders on IS spec document
[RRAPIS-88] - Draft wireframes for provenance system and workflows
[RRAPIS-89] - Draft wireframes for datastore system and workflows
[RRAPIS-90] - Draft wireframes for decision register and workflows
[RRAPIS-91] - Draft implementation options for decision register
[RRAPIS-92] - Draft implementation options for datastore
[RRAPIS-94] - Draft specification section for access to code
[RRAPIS-96] - Revise Cloud costing based on specification document and other requirements
[RRAPIS-99] - Action items from provenance implementation options meeting
[RRAPIS-103] - Finalise spec doc draft
[RRAPIS-114] - Apply assessment criteria to provenance implementation options
[RRAPIS-115] - Apply assessment criteria against implementation options
[RRAPIS-116] - Apply implementation criteria for Decision Register
[RRAPIS-117] - Apply implementation criteria for Data Store
[RRAPIS-118] - Detailed specification doc: Landing pages specification content
[RRAPIS-119] - Detailed specification doc: User access and management specification content
[RRAPIS-120] - Detailed specification doc: Identity framework specification content
[RRAPIS-125] - Consolidate data store spec
[RRAPIS-127] - Add decision register implementation option assessment tables to detailed spec doc
[RRAPIS-128] - Add provenance store implementation option assessment tables to detailed spec doc
[RRAPIS-152] - Check with the IPMF team about their data requirements estimates and refine if needed
[RRAPIS-191] - Data store authorisation model
[RRAPIS-198] - Create Dataset UI component using JSON Schema
[RRAPIS-204] - Test authorisation flow
[RRAPIS-211] - As a modeller, I want to upload a relevant dataset via the AWS CLI pathway using prepared scripts
[RRAPIS-230] - Document dataset registry workflow
[RRAPIS-231] - Document access control/authorisation workflow
[RRAPIS-245] - RRAPIS Make the icons and HTML-reset stylesheet be available to anything as static content
[RRAPIS-248] - Implement a lookup for usage licence in data store form
[RRAPIS-257] - Usability testing: As a user, I want to learn about the RRAP IS system v0.1 release and what I can do in the system
[RRAPIS-259] - Create release notes for v0.1 Anemone release
[RRAPIS-261] - Check if CSS in static site can be / is used in landing portal
[RRAPIS-263] - Usability: Carry out a usability test on dataset registration
[RRAPIS-264] - Scope out some Data Store user stories to test
[RRAPIS-266] - Discuss how to manage static resources for RRAP IS
[RRAPIS-267] - Doc Registering and uploading a dataset
[RRAPIS-280] - Button to request login authorisation
[RRAPIS-307] - Create a user account for each developer on dev and stage
[RRAPIS-317] - Add Landing portal API extensions to capture user requests
[RRAPIS-321] - File upload and download speed benchmark to data store/s3
[RRAPIS-324] - Create a 'Welcome' page for new users
[RRAPIS-330] - Allow users to view request history in User Profile > Roles
[RRAPIS-332] - Improve presentation of data store dataset view page in UI
[RRAPIS-338] - Front page of data store - remove news items and replace with user doc links
[RRAPIS-342] - Improve upload page instructions
[RRAPIS-344] - Dataset item page breadcrumb - use text alternative to Home
[RRAPIS-360] - Add content for "TODO - What is M&DS and its role" in GH page site
[RRAPIS-361] - User doc on usage licences available
[RRAPIS-362] - User doc on Maximum file size
[RRAPIS-363] - User doc on "How do I upload dataset files?"
[RRAPIS-364] - user doc on "Downloading and sync datasets"
[RRAPIS-365] - User doc on "Viewing a dataset"
[RRAPIS-366] - User doc on v1.0 data store metadata
[RRAPIS-367] - Update FAQ
[RRAPIS-368] - Data store provides link to roles landing portal menu when unauthorised
[RRAPIS-369] - Make access roles more accurate on data store API
[RRAPIS-374] - Add additional non-cc dataset licences
[RRAPIS-465] - Extend request access documentation to provide detailed recommendations for each role
[RRAPIS-466] - Usability test plan for Landing portal v1.0
[RRAPIS-467] - Integration test: Login to system
[RRAPIS-468] - Change text in Dataset view from "RO-Crate Metadata" to "JSON metadata"
[RRAPIS-470] - Figure out how to implement contact us
[RRAPIS-472] - Remove "Dataset files" section in Data Store
[RRAPIS-479] - Develop data store usability testing running sheet
[RRAPIS-482] - Add Data License to a field in the DataSet UI
[RRAPIS-486] - Create pre-dev staging infrastructure

Sub-task

[RRAPIS-475] - Add FAQ entry to "How do I log a fault?"
[RRAPIS-483] - Adapt data store UI to handle check-read-access/check-write-access
[RRAPIS-484] - Add read only AWS cred generation for downloading datasets