Sec­ond twit­ter thread about fol­low­ing a sim­ple ex­am­ple to learn Apache Air­flow.

So, let's talk a bit more about watch­ing TV us­ing Apache Air­flow (which is not a very good idea legal­ly, but has good po­ten­tial tech­ni­cal­ly!)

This is the first thread: http­s://t.­co/KG8VgqkQ­TO

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

When we last looked, we had two air­flow tasks look­ing for the lat­est episodes of two se­ries, and one task that was us­ing xcom_pull to get that in­for­ma­tion and find tor­rents.

Well, that was not re­al­ly work­ing well.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Why? Be­cause of this line in the down­stream task. pic.twit­ter.­com/vUKL­cw584O

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

So, xcom_pull "pull­s" da­ta that was "pushed" by the up­stream tasks. But ... we are pulling once. If two tasks pushed, then we will on­ly pull the lat­est val­ue, and search for a sin­gle se­ries' episode.

So, we have to do bet­ter.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Turns out xcom_pull can take a "task_id­s" ar­gu­men­t, which if it's a list of tasks will pull the val­ue re­quired from EV­ERY ONE of those tasks.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Luck­i­ly, op­er­a­tors know the ids of their up­stream (and down­stream) tasks, so this is easy to do.

So, I can get all that data, it­er­ate, and process each one as be­fore. pic.twit­ter.­com/VeB­BI­jY­OqA

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

An­oth­er in­ter­est­ing bit is that there is no need to ex­plic­it­ly de­clare each "search for lat­est episode of se­ries X" task in the DAG. That's what we have con­fig files and loops for (no­tice the use of the dag as con­text man­ager) pic.twit­ter.­com/dE­sieHI­H8e

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

So, now we have a WORK­ING thing that giv­en a list of se­ries names ends up with a list of tor­rents and se­ries meta­da­ta.

The next log­i­cal step is to down­load those things some­how.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

One way to do this is:

1) Start trans­mis­sion-­dae­mon
2) Con­fig­ure how­ev­er you want
3) Let's use trans­mis­sion-re­mote to add tor­rents to it

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Since trans­mis­sion-re­mote is just a com­mand, we can use Python­Op­er­a­tor's more rus­tic broth­er Bash­Op­er­a­tor

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Ba­si­cal­ly, Bash­Op­er­a­tor runs com­mand­s.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Here is what works (will ex­plain) pic.twit­ter.­com/i­AmF­DiAoEK

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

DA­FUQ, you may say. And in­deed.

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

But ba­si­cal­ly: Bash­Op­er­a­tor is **tem­plat­ed**. That string there? It's ran through jin­ja with the con­text (re­mem­ber the con­tex­t?) and thus things are re­placed.

So it ends up be­ing some­thing like this (but much longer) be­cause it's ex­tract­ing the tor­rents from the con­tex­t. pic.twit­ter.­com/oZB­MU3n­Drs

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

So, our DAG now looks like this. pic.twit­ter.­com/hXW­bkp­bKlx

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

And what hap­pens when you trig­ger that DAG? pic.twit­ter.­com/kahGmSVT­pj

— Rober­to Alsi­na (@ralsi­na) February 18, 2020

Com­ing in part 3:

* So, what can you do with that?
* What val­ue is air­flow adding?

— Rober­to Alsi­na (@ralsi­na) February 18, 2020