Configuration files

• Última modificación: Mayo 14, 2022

%cd /tmp
!rm -rf output/ all_sales.csv
!mkdir -p input/

/tmp

%%writefile /tmp/luigi.cfg
[DownloadSalesData]
input_folder=input

Overwriting /tmp/luigi.cfg

%%writefile input/sales1.csv
Fance,May,100
NA
Germany,May,200

Overwriting input/sales1.csv

%%writefile input/sales2.csv
Fance,June,140
Germany,June,180
UK,June,180

Overwriting input/sales2.csv

%%writefile input/sales3.csv
France,June,240
None
Germany,June,180

Overwriting input/sales3.csv

%%writefile __init__.py
#

Overwriting __init__.py

%%writefile pipeline.py

import os

import luigi
from luigi import IntParameter, LocalTarget, Parameter, Task

OUTPUT_FOLDER = "output"


class DownloadFile(Task):

    input_folder = Parameter()
    file_name = Parameter()
    index = IntParameter()

    def output(self):
        output_path = os.path.join(
            OUTPUT_FOLDER,
            str(self.index),
            self.file_name,
        )
        return LocalTarget(output_path)

    def run(self):
        input_path = os.path.join(
            self.input_folder,
            self.file_name,
        )
        with open(input_path) as in_file:
            with self.output().open("w") as out_file:
                for line in in_file:
                    if "," in line:
                        out_file.write(line)


class DownloadSalesData(Task):
    input_folder = Parameter()

    def output(self):
        return LocalTarget("all_sales.csv")

    def run(self):
        processed_files = []
        counter = 1
        for file in sorted(os.listdir(self.input_folder)):
            target = yield DownloadFile(
                self.input_folder,
                file,
                counter,
            )
            counter += 1
            processed_files.append(target)

        with self.output().open("w") as out_file:
            for file in processed_files:
                with file.open() as in_file:
                    for line in in_file:
                        out_file.write(line)

Overwriting pipeline.py

!LUIGI_CONFIG_PATH=/tmp/luigi.cfg python3 -m luigi --module pipeline DownloadSalesData  --local-scheduler

DEBUG: Checking if DownloadSalesData(input_folder=input) is complete
INFO: Informed scheduler that task   DownloadSalesData_input_3d041dae39   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadSalesData(input_folder=input)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) new requirements      DownloadSalesData(input_folder=input)
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Checking if DownloadFile(input_folder=input, file_name=sales1.csv, index=1) is complete
INFO: Informed scheduler that task   DownloadFile_sales1_csv_1_input_c3a4c863ff   has status   PENDING
INFO: Informed scheduler that task   DownloadSalesData_input_3d041dae39   has status   PENDING
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadFile(input_folder=input, file_name=sales1.csv, index=1)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) done      DownloadFile(input_folder=input, file_name=sales1.csv, index=1)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownloadFile_sales1_csv_1_input_c3a4c863ff   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadSalesData(input_folder=input)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) new requirements      DownloadSalesData(input_folder=input)
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Checking if DownloadFile(input_folder=input, file_name=sales2.csv, index=2) is complete
INFO: Informed scheduler that task   DownloadFile_sales2_csv_2_input_83e73fcbf0   has status   PENDING
INFO: Informed scheduler that task   DownloadSalesData_input_3d041dae39   has status   PENDING
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadFile(input_folder=input, file_name=sales2.csv, index=2)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) done      DownloadFile(input_folder=input, file_name=sales2.csv, index=2)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownloadFile_sales2_csv_2_input_83e73fcbf0   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadSalesData(input_folder=input)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) new requirements      DownloadSalesData(input_folder=input)
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Checking if DownloadFile(input_folder=input, file_name=sales3.csv, index=3) is complete
INFO: Informed scheduler that task   DownloadFile_sales3_csv_3_input_18bd499a8c   has status   PENDING
INFO: Informed scheduler that task   DownloadSalesData_input_3d041dae39   has status   PENDING
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadFile(input_folder=input, file_name=sales3.csv, index=3)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) done      DownloadFile(input_folder=input, file_name=sales3.csv, index=3)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownloadFile_sales3_csv_3_input_18bd499a8c   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) running   DownloadSalesData(input_folder=input)
INFO: [pid 890] Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) done      DownloadSalesData(input_folder=input)
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownloadSalesData_input_3d041dae39   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
INFO: Worker Worker(salt=049381492, workers=1, host=b148b7d2dd75, username=root, pid=890) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 4 tasks of which:
* 4 ran successfully:
    - 3 DownloadFile(input_folder=input, file_name=sales1.csv, index=1) ...
    - 1 DownloadSalesData(input_folder=input)

This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

!cat all_sales.csv

Fance,May,100
Germany,May,200
Fance,June,140
Germany,June,180
UK,June,180
France,June,240
Germany,June,180

!ls -1 /tmp/output

1
2
3

!ls -1 /tmp/output/1/

sales1.csv

!ls -1 /tmp/output/2/

sales2.csv

!ls -1 /tmp/output/3/

sales3.csv

!cat /tmp/output/1/sales1.csv

Fance,May,100
Germany,May,200