Librería re (expresiones regulares en Python) — 28:07 min

• 28:07 min | Última modificación: Octubre 5, 2021 | YouTube

Preparación de datos

## Se crea el directorio de entrada
!rm -rf input output
!mkdir input

%%writefile input/text0.txt
Analytics is the discovery, interpretation, and communication of meaningful patterns
in data. Especially valuable in areas rich with recorded information, analytics relies
on the simultaneous application of statistics, computer programming and operations research
to quantify performance.

Organizations may apply analytics to business data to describe, predict, and improve business
performance. Specifically, areas within analytics include predictive analytics, prescriptive
analytics, enterprise decision management, descriptive analytics, cognitive analytics, Big
Data Analytics, retail analytics, store assortment and stock-keeping unit optimization,
marketing optimization and marketing mix modeling, web analytics, call analytics, speech
analytics, sales force sizing and optimization, price and promotion modeling, predictive
science, credit risk analysis, and fraud analytics. Since analytics can require extensive
computation (see big data), the algorithms and software used for analytics harness the most
current methods in computer science, statistics, and mathematics.

Writing input/text0.txt

%%writefile input/text1.txt
The field of data analysis. Analytics often involves studying past historical data to
research potential trends, to analyze the effects of certain decisions or events, or to
evaluate the performance of a given tool or scenario. The goal of analytics is to improve
the business by gaining knowledge which can be used to make improvements or changes.

Writing input/text1.txt

%%writefile input/text2.txt
Data analytics (DA) is the process of examining data sets in order to draw conclusions
about the information they contain, increasingly with the aid of specialized systems
and software. Data analytics technologies and techniques are widely used in commercial
industries to enable organizations to make more-informed business decisions and by
scientists and researchers to verify or disprove scientific models, theories and
hypotheses.

Writing input/text2.txt

import glob

raw_text = []
for filename in glob.glob("input/*.txt"):
    with open(filename, "r") as f:
        raw_text += f.readlines()

raw_text = " ".join(raw_text)
raw_text = raw_text.replace("\n", "")

phrases = raw_text.split(".")
phrases = [t.strip() + "." for t in phrases]
phrases

['Analytics is the discovery, interpretation, and communication of meaningful patterns in data.',
 'Especially valuable in areas rich with recorded information, analytics relies on the simultaneous application of statistics, computer programming and operations research to quantify performance.',
 'Organizations may apply analytics to business data to describe, predict, and improve business performance.',
 'Specifically, areas within analytics include predictive analytics, prescriptive analytics, enterprise decision management, descriptive analytics, cognitive analytics, Big Data Analytics, retail analytics, store assortment and stock-keeping unit optimization, marketing optimization and marketing mix modeling, web analytics, call analytics, speech analytics, sales force sizing and optimization, price and promotion modeling, predictive science, credit risk analysis, and fraud analytics.',
 'Since analytics can require extensive computation (see big data), the algorithms and software used for analytics harness the most current methods in computer science, statistics, and mathematics.',
 'The field of data analysis.',
 'Analytics often involves studying past historical data to research potential trends, to analyze the effects of certain decisions or events, or to evaluate the performance of a given tool or scenario.',
 'The goal of analytics is to improve the business by gaining knowledge which can be used to make improvements or changes.',
 'Data analytics (DA) is the process of examining data sets in order to draw conclusions about the information they contain, increasingly with the aid of specialized systems and software.',
 'Data analytics technologies and techniques are widely used in commercial industries to enable organizations to make more-informed business decisions and by scientists and researchers to verify or disprove scientific models, theories and hypotheses.',
 '.']

words = raw_text.split()
words[:20]

['Analytics',
 'is',
 'the',
 'discovery,',
 'interpretation,',
 'and',
 'communication',
 'of',
 'meaningful',
 'patterns',
 'in',
 'data.',
 'Especially',
 'valuable',
 'in',
 'areas',
 'rich',
 'with',
 'recorded',
 'information,']

#
# Uso de `in`
# =============================================================================
#
set(word for word in words if "ly" in word)

{'Analytics',
 'Analytics,',
 'Especially',
 'Specifically,',
 'analysis,',
 'analysis.',
 'analytics',
 'analytics,',
 'analytics.',
 'analyze',
 'apply',
 'increasingly',
 'widely'}

#
# Uso de re.search
# =============================================================================
# Obtiene el primer match con la cadena de busqueda
#
import re

g = re.search("ly", raw_text)
g

<_sre.SRE_Match object; span=(3, 5), match='ly'>

g.group(0)

'ly'

re.search("analytics", raw_text, re.IGNORECASE)

<_sre.SRE_Match object; span=(0, 9), match='Analytics'>

re.search("analytics", raw_text, re.IGNORECASE).group(0)

'Analytics'

Resumen de operadores usados en expresiones regulares

.          Wildcard, matches any character
^abc       Matches some pattern abc at the start of a string
abc$       Matches some pattern abc at the end of a string
[abc]      Matches one of a set of characters
[A-Z0-9]   Matches one of a range of characters
ed|ing|s   Matches one of the specified strings (disjunction)
*          Zero or more of previous item, e.g. a*, [a-z]* (also known as Kleene Closure)
+          One or more of previous item, e.g. a+, [a-z]+
?          Zero or one of the previous item (i.e. optional), e.g. a?, [a-z]?
{n}        Exactly n repeats where n is a non-negative integer
{n,}       At least n repeats
{,n}       No more than n repeats
{m,n}      At least m and no more than n repeats
a(b|c)+    Parentheses that indicate the scope of the operators

Principales caracteres usados en expresiones regulares

\b   Word boundary (zero width)
\d   Any decimal digit (equivalent to [0-9])
\D   Any non-digit character (equivalent to [^0-9])
\s   Any whitespace character (equivalent to [ \t\n\r\f\v])
\S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
\w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
\W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
\t   The tab character
\n   The newline character

#
# Uso de findall
# =============================================================================
# Extrae las palabras terminadas en `ing`
# Note la r en la expresión regular
#
# \b   Word boundary (zero width)
#
sorted(set(re.findall(r"\b[a-zA-Z]+ing\b", raw_text)))

['examining',
 'gaining',
 'keeping',
 'marketing',
 'modeling',
 'programming',
 'sizing',
 'studying']

#
# Explique el resultado de esta expresión regular
# =============================================================================
# \b   Word boundary (zero width)
# \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
#
sorted(set(re.findall(r"\b\S+ing\b", raw_text)))

['examining',
 'gaining',
 'marketing',
 'modeling',
 'programming',
 'sizing',
 'stock-keeping',
 'studying']

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
#
sorted(set(re.findall(r"\b[a-zA-Z\-]+ing\b", raw_text)))

['examining',
 'gaining',
 'marketing',
 'modeling',
 'programming',
 'sizing',
 'stock-keeping',
 'studying']

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
#
re.findall(r"\btwo three\b", "one two three four five six")

['two three']

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
#
re.findall(r"\btwo three\b", "one two  three four five six")

[]

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\btwo\Wthree\b", "one two three four five six")

['two three']

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\btwo\Wthree\b", "one two  three four five six")

[]

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\btwo\W+three\b", "one two  three four five six")

['two  three']

#
# Explique el resultado de esta expresión regular:
# =============================================================================
# \b   Word boundary (zero width)
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\bthree\W+two\b|\btwo\W+three\b", "one two  three four five six")

['two  three']

#
# Palabras cercanas
# =============================================================================
# Dos palabras separadas por otra cualquiera
#
# \b   Word boundary (zero width)
# \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\btwo\W+\w+\W+four\b", "one two three four five six")

['two three four']

#
# ?: indica que no se desea extraer la expresión reconocida como tal
# =============================================================================
# Palabras cercanas
# Dos palabras separadas por una, dos o tres palabras no especificadas
#
# \b   Word boundary (zero width)
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.findall(r"\btwo\W+(?:\w+\W+){1,3}five", "one two three four five six")

['two three four five']

#
# (?=...)   Aserción hacia adelante
# =============================================================================
# Match Isaac si es seguido de Asimov
#
# \b   Word boundary (zero width)
# \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
#
re.findall(r"\b\S+ (?=Asimov)", "Carl Sagan Isaac Asimov James Bond")

['Isaac ']

#
# (?!...)   Aserción negativa hacia adelante
# =============================================================================
# Match si la cadena no es seguida de Asimov
#
# \b   Word boundary (zero width)
# \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
#
re.findall(r"\b\S+ (?!Asimov)", "Carl Sagan Isaac Asimov James Bond")

['Carl ', 'Sagan ', 'Asimov ', 'James ']

#
# (?<=...)   Precedencia
# =============================================================================
# \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
#
re.findall(r"(?<=-)\w+", "sun right-hand moon left-hand")

['hand', 'hand']

#
# (?<!...)   Precedencia
# =============================================================================
#
re.findall(r"(?<!-)\w+", "sun right-hand moon left-hand")

['sun', 'right', 'and', 'moon', 'left', 'and']

#
# Manejo de grupos
# =============================================================================
#
m = re.match("([abc])+", "abc")
m.groups()

('c',)

re_ = re.compile("(a(b)c)d")
m = re_.match("abcd")
m.group(0)

'abcd'

m.group(1)

'abc'

m.group(2)

'b'

m.group(2, 1, 0)

('b', 'abc', 'abcd')

#
# (?P<word>...) Se asigna el nombre de <word> a la parte reconocida
# =============================================================================
#
p = re.compile(r"(?P<word>\b\w+\b)")
m = p.search("(((( Lots of punctuation )))")
m.group("word")

'Lots'

m.group(1)

'Lots'

#
# Extracción de los grupos reconocidos mediante un diccionario
# =============================================================================
# \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
# (?P<word>...) Se asigna el nombre de <word> a la parte reconocida
#
m = re.match(r"(?P<first>\w+) (?P<last>\w+)", "Jane Doe")
m.groupdict()

{'first': 'Jane', 'last': 'Doe'}

#
# Compilación de expresiones regulares
# =============================================================================
# \b   Word boundary (zero width)
# \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
# (?P<word>...) Se asigna el nombre de <word> a la parte reconocida
#
re_ = re.compile(r"\b\S+ing\b")
re_.search(raw_text)

<_sre.SRE_Match object; span=(228, 239), match='programming'>

m = re_.search(raw_text)
m.group()

'programming'

#
# Búsqueda de palabras que comiencen por `ana`
# =============================================================================
# match() determina si la expresión regular está al principio de la cadena
#
# \b   Word boundary (zero width)
# \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
#
re_ = re.compile(r"\b^Ana\S+")
re_.match(raw_text)

<_sre.SRE_Match object; span=(0, 9), match='Analytics'>

re_.match(raw_text).group()

'Analytics'

#
# findall
# =============================================================================
# \d   Any decimal digit (equivalent to [0-9])
#
re_ = re.compile(r"\d+")
re_.findall("12 drummers drumming, 11 pipers piping, 10 lords a-leaping")

['12', '11', '10']

#
# finditer
# =============================================================================
#
iterator = re_.finditer("12 drummers drumming, 11 10 ...")
iterator

<callable_iterator at 0x7fd4237d80f0>

for match in iterator:
    print(match.span())

(0, 2)
(22, 24)
(25, 27)

#
# Se asigna el nombre de <word> a la parte reconocida
# =============================================================================
# \b   Word boundary (zero width)
# \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
# (?P<word>...) Se asigna el nombre de <word> a la parte reconocida
#
p = re.compile(r"(?P<word>\b\w+\b)")
m = p.search("(((( Lots of punctuation )))")
m.group("word")

'Lots'

m.group(1)

'Lots'

#
# Extracción de los grupos reconocidos mediante un diccionario
# =============================================================================
# \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
# (?P<word>...) Se asigna el nombre de <word> a la parte reconocida
#
m = re.match(r"(?P<first>\w+) (?P<last>\w+)", "Jane Doe")
m.groupdict()

{'first': 'Jane', 'last': 'Doe'}

#
# Stemmer
#
def stemmer(word):
    for suffix in ["ing", "ly", "ed", "ious", "ies", "ive", "es", "s", "ment"]:
        if word.endswith(suffix):
            return word[: -len(suffix)]
    return word


stemmer("computing")

'comput'

import re

re.findall(r"^.*(ing|ly|ed|ious|ies|ive|es|s|ment)$", "computing")

['ing']

re.findall(r"^.*(?:ing|ly|ed|ious|ies|ive|es|s|ment)$", "computing")

['computing']

re.findall(r"^(.*)(ing|ly|ed|ious|ies|ive|es|s|ment)$", "processes")

[('processe', 's')]

re.findall(r"^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)$", "processes")

[('process', 'es')]

#
# Tokenización usando re
#
raw = """'When I'M a Duchess,' she said to herself, (not in a very hopeful tone
though), 'I won't have any pepper in my kitchen AT ALL. Soup does very
well without--Maybe it's always pepper that makes people hot-tempered,'..."""

re.split(r" ", raw)

["'When",
 "I'M",
 'a',
 "Duchess,'",
 'she',
 'said',
 'to',
 'herself,',
 '(not',
 'in',
 'a',
 'very',
 'hopeful',
 'tone\nthough),',
 "'I",
 "won't",
 'have',
 'any',
 'pepper',
 'in',
 'my',
 'kitchen',
 'AT',
 'ALL.',
 'Soup',
 'does',
 'very\nwell',
 'without--Maybe',
 "it's",
 'always',
 'pepper',
 'that',
 'makes',
 'people',
 "hot-tempered,'..."]

#
# \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
#
re.split(r"\W+", raw)

['',
 'When',
 'I',
 'M',
 'a',
 'Duchess',
 'she',
 'said',
 'to',
 'herself',
 'not',
 'in',
 'a',
 'very',
 'hopeful',
 'tone',
 'though',
 'I',
 'won',
 't',
 'have',
 'any',
 'pepper',
 'in',
 'my',
 'kitchen',
 'AT',
 'ALL',
 'Soup',
 'does',
 'very',
 'well',
 'without',
 'Maybe',
 'it',
 's',
 'always',
 'pepper',
 'that',
 'makes',
 'people',
 'hot',
 'tempered',
 '']

Extracción de direcciones web de un archivo

!wget --quiet {"https://raw.githubusercontent.com/jdvelasq/datalabs/master/datasets/short_tweets.csv"} -P /tmp/

import re

import pandas as pd

short_tweets = pd.read_csv("/tmp/short_tweets.csv")
text = short_tweets.text

links = []
users = []
dates = []
hashtags = []
textfiles = []

for tweet in text:

    # \b   Word boundary (zero width)
    # \d   Any decimal digit (equivalent to [0-9])
    # \D   Any non-digit character (equivalent to [^0-9])
    # \s   Any whitespace character (equivalent to [ \t\n\r\f\v])
    # \S   Any non-whitespace character (equivalent to [^ \t\n\r\f\v])
    # \w   Any alphanumeric character (equivalent to [a-zA-Z0-9_])
    # \W   Any non-alphanumeric character (equivalent to [^a-zA-Z0-9_])
    # \t   The tab character
    # \n   The newline character

    link = re.findall(r"http\S+", tweet)
    if link:
        links.append(link)

    user = re.findall(r"@\w+", tweet)
    if user:
        users.append(user)

    date = re.findall(r"\d{1,2}\s\w+\sago", tweet)
    if date:
        dates.append(date)

    hashtag = re.findall(r"#\w+", tweet)
    if hashtag:
        hashtags.append(hashtag)

    textfile = re.findall(r"^[aeiouAEIOU]{2,3}.+txt", text)
    if textfile:
        textfiles.append(textfile)


display(
    links[:5],
    users[:5],
    dates[:5],
    hashtags[:5],
    textfiles[:5],
)

[['http://twitpic.com/2y2es'],
 ['http://apps.facebook.com/dogbook/profile/view/5248435'],
 ['http://tr.im/imji'],
 ['http://apps.facebook.com/dogbook/profile/view/6176014'],
 ['http://ff.im/1XTTi']]

[['@andywana'], ['@oanhLove'], ['@BatManYNG'], ['@Starrbby'], ['@katortiz']]

[['32 minutes ago'],
 ['3 hours ago'],
 ['2 hours ago'],
 ['4 months ago'],
 ['2 hours ago']]

[['#itm'], ['#therapyfail'], ['#fb'], ['#TTSC', '#24'], ['#gayforpeavy']]