BEGIN:VCALENDAR
VERSION:2.0
PRODID:icalendar-ruby
CALSCALE:GREGORIAN
BEGIN:VEVENT
DTSTAMP:20260624T161752Z
UID:2a4e4bfc-8ffa-4d95-8086-3a31dd8f7619
DTSTART:20250108T120000Z
DTEND:20250108T150000Z
DESCRIPTION:This workshop will build on the half-day workshop ["Building Sc
 alable and Maintainable Data Pipelines with Omnipy (Part 1 - Beginner leve
 l)](https://tess.elixir-europe.org/events/building-scalable-and-maintainab
 le-data-pipelines-with-omnipy-part-1) we are holding before lunch.\n\nIn t
 his second workshop\, participants will learn how to develop various types
  of data flows in Omnipy\, including integration with web services. They w
 ill make use of the powerful industry-developed Prefect orchestration engi
 ne to scale up the game and deploy high-throughput ETL flows using externa
 l compute resources.\n\nThe workshop is divided into three parts:\n\n1. Th
 e first part will introduce the slogan "parse\, don't validate" and show h
 ow these concepts are implemented in Omnipy. On this background\, we will 
 introduce the three types of data flows supported by Omnipy: linear\, DAG\
 , and function flows. We will also\, through hands-on examples\, show how 
 to make use of various job modifiers to power up and customise predefined 
 tasks and flows to construct more complex data flows.\n1. The second part 
 will focus on integrating data flows with web services through REST APIs. 
 We will mainly focus on extracting data from data sources\, but will also 
 touch upon loading results onto data sinks. Hands-on examples will introdu
 ce tasks and flows that allow flattening of JSON data into relational tabu
 lar form for mapping\, and then restructuring the results back to JSON.\n1
 . The last part will introduce Omnipy's integration with S3-based cloud st
 orage and the Prefect ETL orchestration library. As a hands-on exercise\, 
 the participant will scale up the data flow developed in the second part o
 f the workshop by deploying it on an external compute infrastructure\, pot
 entially the Kubernetes-based NIRD Toolkit from SIGMA2 (if Prefect-integra
 tion in NIRD is finalised in time for the workshop).
LOCATION:Elektronisk klasserom\, Universitetsbiblioteket i Oslo\, 39 Moltke
  Moes vei
SUMMARY:Building Scalable and Maintainable Data Pipelines with Omnipy (Part
  2)
URL;VALUE=URI:https://www.ub.uio.no/english/courses-events/events/dsc/2025/
 digital-scholarship-days/11-building-scalable-and-maintainable-data-pipeli
 nes-with-omnipy-part2
END:VEVENT
END:VCALENDAR