View event

Date: 12 January 2023 @ 12:00 - 16:00

Timezone: Brussels

Duration: 3 hours

Loading map...

Researchers often need to extract, manipulate and integrate data and/or metadata from different sources, such as repositories, databases, or flat files. Much research time is spent on trivial and not-so-trivial details of data wrangling: to reformat data structures; clean up errors; remove duplicate data; or map and integrate dataset fields. Software for data wrangling and analysis, such as Pandas, R or Frictionless, is useful, but researchers still regularly end up with hard-to-reuse scripts, often with manual steps.

uniFAIR is a new Python library with a systematic and scalable approach to research data wrangling. With uniFAIR, researchers can import (meta)data in almost any shape or form: nested JSON; tabular (relational) data; binary streams; or other data structures. Data is continuously parsed and reshaped through a step-by-step process according to a series of data model transformations. uniFAIR provides a catalogue of generic task and subflow templates that the researcher can refine and apply to carry out the transformations needed to wrangle data into the required shape.

For large datasets, uniFAIR allows local test jobs on sample-sized data to be seamlessly scaled up to the full datasets and offloaded to external compute resources. Persistent access to the state of the data is available at every step.

Venue: Georg Sverdrups hus, 39 Moltke Moes vei

City: Oslo

Region: Oslo kommune

Country: Norway

Postcode: 0851

Prerequisites:

The participant should have experience with Python programming. Experience with type hints in Python is useful, but not required.

Learning objectives:

Use type hints in Python in general and to define data models in uniFAIR/Pydantic
Understand the ideas behind the slogan "parse, don't validate"
Know the architecture of uniFAIR and its main classes, and have an overview of the different modules and their usage
Define, refine, apply and revise tasks and flows in uniFAIR - Import data from external REST APIs and flat files
Develop data transformation flows to solve a selection of use cases
Inspect data after each transformation step. Make informed choices on how to configure the next tasks.
Transform nested JSON output into normalized tables (without duplicate data) *Map (meta)data fields from the input data model to the user-defined output model

These outcomes will be demonstrated and not hands-on due to time constraints:

Scale up the data import from a representative sample to a large dataset and deploy the flow on external compute resources (e.g. NIRD service platform)
Orchestrate flow runs using the Prefect web-based GUI and inspect data output from external runs
Get started with contributing to the Open Source catalogue of uniFAIR modules

Organizer: The workshop is provided by ELIXIR Oslo as part of an extended event organised by Digital Scholarship Center (DSC), USIT, Carpentry@UiO, Coderefinery, dScience, Simula, Data Managers Network at UiO, ELIXIR Oslo, Norway's Reproducibility Network (norrn) and University of Oslo Library

Host institutions: University of Oslo

Target audience: PhD, Postdoctoral Fellows, Researchers, Engineers

Capacity: 20

Event types: