Hi,

In this blog, we will discuss about airflow architecture.

The main components of airflow are:

Web Server: User interacts with web server. It is the UI of Airflow, that can be used to get an overview of DAG and tasks, states.

The Web Server has ability to manage users…

Hi, Welcome back.

In previous part, https://jyotisachdeva57.medium.com/part-2-basic-terminologies-in-apache-airflow-1c060a638970 we discussed the basic terminologies in airflow.

In this part of tutorial, we will discuss how to run airflow 2.0 locally.

There are two ways to install airflow locally:

1. PIP https://airflow.apache.org/docs/apache-airflow/stable/start/local.html

2. Docker

We will install latest airflow 2.0 via docker.

Airflow…

We face situations daily where we have to choose between pull and rebase to update the local code with the origin. We will see the difference using an example.

Let’s say we have a master branch and it has only one file Demo.txt.

We add m1 to it and commit…

In this blog, we are going to see the use of Either in scala.

We use Options in scala but why do we want to go for Either?

Either is a better approach in the respect that if something fails we can track down the reason, which in Option None…

In this blog, we will see how to read the Avro files using Flink.

Before reading the files, let’s get an overview of Flink.

There are two types of processing — batch and real-time.

  • Batch Processing: Processing based on the data collected over time.
  • Real-time Processing: Processing based on immediate data for an instant result.

Real-time processing is in demand and Apache Flink is…

In this blog, I’m going to discuss currying and partially applied functions.

CURRYING

Currying splits method with multiple parameters into a chain of functions each with one parameter.

Let’s understand currying using an example:

scala> def multiply(a: Int)(b: Int)(c: Int) = a * b * c

is the same as:

Jyoti Sachdeva

Data Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store