Conteo de palabras con mrjob

• Última modificación: Mayo 13, 2022

Archivos de prueba

#
# Se crea el directorio de entrada
#
!rm -rf /tmp/wordcount
!mkdir -p /tmp/wordcount/input
%cd /tmp/wordcount
!ls

/tmp/wordcount
input

%%writefile input/text0.txt
Analytics is the discovery, interpretation, and communication of meaningful patterns
in data. Especially valuable in areas rich with recorded information, analytics relies
on the simultaneous application of statistics, computer programming and operations research
to quantify performance.

Organizations may apply analytics to business data to describe, predict, and improve business
performance. Specifically, areas within analytics include predictive analytics, prescriptive
analytics, enterprise decision management, descriptive analytics, cognitive analytics, Big
Data Analytics, retail analytics, store assortment and stock-keeping unit optimization,
marketing optimization and marketing mix modeling, web analytics, call analytics, speech
analytics, sales force sizing and optimization, price and promotion modeling, predictive
science, credit risk analysis, and fraud analytics. Since analytics can require extensive
computation (see big data), the algorithms and software used for analytics harness the most
current methods in computer science, statistics, and mathematics.

Writing input/text0.txt

%%writefile input/text1.txt
The field of data analysis. Analytics often involves studying past historical data to
research potential trends, to analyze the effects of certain decisions or events, or to
evaluate the performance of a given tool or scenario. The goal of analytics is to improve
the business by gaining knowledge which can be used to make improvements or changes.

Writing input/text1.txt

%%writefile input/text2.txt
Data analytics (DA) is the process of examining data sets in order to draw conclusions
about the information they contain, increasingly with the aid of specialized systems
and software. Data analytics technologies and techniques are widely used in commercial
industries to enable organizations to make more-informed business decisions and by
scientists and researchers to verify or disprove scientific models, theories and
hypotheses.

Writing input/text2.txt

Implementación del Algoritmo MapReduce

%%writefile word_counter.py

import string

from mrjob.job import MRJob


class WordCounter(MRJob):
    def preprocessing(self, word):
        word = word.lower()
        word = word.translate(str.maketrans("", "", string.punctuation))
        word = word.replace("\n", "")
        return word

    def mapper(self, _, line):
        for word in line.split():
            word = self.preprocessing(word)
            yield word, 1

    def reducer(self, key, values):
        yield key, sum(values)


if __name__ == "__main__":
    WordCounter.run()

Writing word_counter.py

Ejecución

!python3 word_counter.py  input/*.txt

No configs found; falling back on auto-configuration
No configs specified for inline runner
Creating temp directory /tmp/word_counter.root.20220526.023642.466507
Running step 1 of 1...
job output is in /tmp/word_counter.root.20220526.023642.466507/output
Streaming final output from /tmp/word_counter.root.20220526.023642.466507/output...
"business"      4
"by"    2
"call"  1
"can"   2
"draw"  1
"effects"       1
"enable"        1
"enterprise"    1
"especially"    1
"evaluate"      1
"events"        1
"examining"     1
"a"     1
"about" 1
"aid"   1
"algorithms"    1
"analysis"      2
"analytics"     20
"moreinformed"  1
"most"  1
"of"    8
"often" 1
"extensive"     1
"field" 1
"for"   1
"force" 1
"fraud" 1
"gaining"       1
"given" 1
"goal"  1
"harness"       1
"decision"      1
"decisions"     2
"describe"      1
"descriptive"   1
"discovery"     1
"disprove"      1
"certain"       1
"changes"       1
"cognitive"     1
"commercial"    1
"communication" 1
"computation"   1
"computer"      2
"involves"      1
"is"    3
"knowledge"     1
"make"  2
"management"    1
"marketing"     2
"application"   1
"apply" 1
"are"   1
"areas" 2
"assortment"    1
"be"    1
"big"   2
"analyze"       1
"and"   15
"include"       1
"increasingly"  1
"industries"    1
"information"   2
"interpretation"        1
"theories"      1
"they"  1
"to"    12
"which" 1
"widely"        1
"with"  2
"within"        1
"on"    1
"operations"    1
"optimization"  3
"or"    5
"mathematics"   1
"may"   1
"meaningful"    1
"methods"       1
"mix"   1
"modeling"      2
"models"        1
"potential"     1
"predict"       1
"predictive"    2
"prescriptive"  1
"price" 1
"process"       1
"specialized"   1
"specifically"  1
"speech"        1
"statistics"    2
"stockkeeping"  1
"historical"    1
"hypotheses"    1
"improve"       2
"improvements"  1
"in"    5
"store" 1
"studying"      1
"systems"       1
"techniques"    1
"technologies"  1
"the"   12
"order" 1
"organizations" 2
"past"  1
"patterns"      1
"performance"   3
"tool"  1
"trends"        1
"unit"  1
"used"  3
"valuable"      1
"verify"        1
"web"   1
"researchers"   1
"retail"        1
"rich"  1
"risk"  1
"sales" 1
"scenario"      1
"science"       2
"conclusions"   1
"contain"       1
"credit"        1
"current"       1
"da"    1
"data"  9
"scientific"    1
"scientists"    1
"see"   1
"sets"  1
"simultaneous"  1
"since" 1
"sizing"        1
"software"      2
"programming"   1
"promotion"     1
"quantify"      1
"recorded"      1
"relies"        1
"require"       1
"research"      2
Removing temp directory /tmp/word_counter.root.20220526.023642.466507...

!python3 word_counter.py  input/*.txt > result.txt

No configs found; falling back on auto-configuration
No configs specified for inline runner
Creating temp directory /tmp/word_counter.root.20220526.023643.236983
Running step 1 of 1...
job output is in /tmp/word_counter.root.20220526.023643.236983/output
Streaming final output from /tmp/word_counter.root.20220526.023643.236983/output...
Removing temp directory /tmp/word_counter.root.20220526.023643.236983...

!head result.txt

"business"      4
"by"    2
"call"  1
"can"   2
"draw"  1
"effects"       1
"enable"        1
"enterprise"    1
"especially"    1
"evaluate"      1