When to use lambdas

Monkey patching in Python

Multi-threading is available in Python but it;s often not a good idea

PAY $1 OR MORE TO ALSO UNLOCK!

Teach Your Kids to Code: A Parent-Friendly Guide to Python Programming MSRP: $23.95 Already had

Invent Your Own Computer Games with Python, 4th Edition MSRP: $23.95 Already had

Black Hat Python: Python Programming for Hackers and Pentesters MSRP: $27.95 https://www.amazon.com/Black-Hat-Python-Programming-Pentesters-ebook/dp/B00QL616DW/

Gray Hat Python: Python Programming for Hackers and Reverse Engineers MSRP: $31.95 https://www.amazon.com/Gray-Hat-Python-Programming-Engineers-ebook/dp/B007V2DNEK/

PAY $8 OR MORE TO ALSO UNLOCK!

Mission Python: Code a Space Adventure Game! MSRP: $23.95

Python for Kids: A Playful Introduction to Programming MSRP: $27.95

Cracking Codes with Python: An Introduction to Building and Breaking Ciphers MSRP: $23.95

Python Playground: Geeky Projects for the Curious Programmer MSRP: $23.95

Math Adventures with Python: An Illustrated Guide to Exploring Math with Code MSRP: $23.95

PAY $15 OR MORE TO ALSO UNLOCK!

Python Crash Course, 2nd Edition: A Hands-On, Project-Based Introduction to Programming MSRP: $31.95

Automate the Boring Stuff with Python, 2nd Edition: Practical Programming for Total Beginners MSRP: $31.95

Python Flash Cards PDF ONLY MSRP: $27.95

Serious Python: Black-Belt Advice on Deployment, Scalability, Testing, and More MSRP: $27.95

Impractical Python Projects: Playful Programming Activities to Make You Smarter

DataForScience/Epidemiology101. (2020). [Jupyter Notebook]. Data For Science. https://github.com/DataForScience/Epidemiology101 (Original work published 2020)

Hot Reloading is what provides a great experience with updating your Dash code inside the Jupyter Notebooks

3 display modes of Dash using Jupyter Notebooks:

1. app.run_server(mode='external')
2. app.run_server(mode='inline')
3. app.run_server(mode='jupyterlab')

Moreover, you can display your Dash result inside a Jupyter Notebook using IPython.display.IFrame with this line:

app.run_server(mode='inline')

How Dash can run inside Jupyter Notebooks

Dash can be tried out inside a Jupyter Notebook right in your browser using binder.

To use Dash in Jupyter Notebooks, you have to import:

from jupyter_dash import JupyterDash

instead of:

import dash

Therefore, all the imports could look like that for a typical Dash app inside a Jupyter Notebook:

import plotly.express as px
from jupyter_dash import JupyterDash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output

A generator signals its termination with an exception (StopIteration). This was already a feature of generators.

The "trick" to make coroutines work, is that this exception is used as a real return value when the coroutine terminates (with a return statement).

The return value is (somehow) incapsulated into the exception that's being raised, and the event loop handles it.

In depth explanation: http://www.dabeaz.com/coroutines/

The event loop in python orchestrate the whole "simulated concurrency" among coroutines.

Deep down, python has a library select that talks very closely with the OS and gets data from sockets. This is actually how the orchestra works really at the bottom layer. https://docs.python.org/3/library/select.html#module-select

in fact, async in python was built on top of the generators (that existed in python since long before).

A generator is a function that can be suspended --yielding a value-- and then resumed.

A key functionality of the generators in python is that when they are resumed they can receive a value back from the code that stopped/resumed them. This translates into the syntax new_value = await coroutine()

Async routines collaborate each with the others by saying when they can be paused. This is why they are called coroutines.

The communication between coroutines happen thanks to the event loop

变量

代码组

多行语句

Python标识符规则。

quick web server from terminal using python3 http module

You can create estimation plots with:

• this website
• R package
• Python package
• MATLAB package

To calculate RFM we need recency value. Firstly, we shall specify the most recent transaction as "today" and then to find the latest transaction of a specific client:

df['Recency'] = (today - df.InvoiceDate)/np.timedelta64(1,'D')

Calculating frequency and aggregating data of each client may be done with the groupby method:

abt = df.groupby(['CustomerID']).agg({'Recency':'min', 'MonetaryValue':'sum', 'InvoiceNo':'count'})

lastly, we can update the column names and display RFM data:

abt = df.groupby(['CustomerID']).agg({'Recency':'min', 'MonetaryValue':'sum', 'InvoiceNo':'count'})
abt.rename(columns = {'InvoiceNo':'Frequency'}, inplace = True)
abt = abt[['Recency', 'Frequency', 'MonetaryValue']]
abt.head()

Deleting rows where value is null:

df = df[~df.CustomerID.isnull()]

Assigning different data types to columns:

df['CustomerID'] = df.CustomerID.astype(int)

Deleting irrelevant columns:

df.drop(['Description', 'StockCode', 'Country'], axis = 1, inplace = True)

Checking the % of missing data:

print(str(round(df.isnull().any(axis=1).sum()/df.shape[0]*100,2))+'% obserwacji zawiera braki w danych.')

Sample output:

24.93% obserwacji zawiera braki w danych.

Building a simple dataframe with a summary of our columns (data types, sum and % of nulls):

summary = pd.DataFrame(df.dtypes, columns=['Dtype'])
summary['Nulls'] = pd.DataFrame(df.isnull().any())
summary['Sum_of_nulls'] = pd.DataFrame(df.isnull().sum())
summary['Per_of_nulls'] = round((df.apply(pd.isnull).mean()*100),2)
summary.Dtype = summary.Dtype.astype(str)
print(summary)

the output:

                      Dtype  Nulls  Sum_of_nulls  Per_of_nulls
InvoiceNo            object  False             0         0.000
StockCode            object  False             0         0.000
Description          object   True          1454         0.270
Quantity              int64  False             0         0.000
InvoiceDate  datetime64[ns]  False             0         0.000
UnitPrice           float64  False             0         0.000
CustomerID          float64   True        135080        24.930
Country              object  False             0         0.000

When to use different programming languages (advice from an Amazon employee):

• Java - enterprise applications
• C# - Microsoft's spin on Java (useful in the Microsoft's ecosystem)
• Ruby - when speed is more important then legibility or debugging
• Python - same as Ruby but also for ML/AI (don't forget to use type hinting to make life a little saner)
• Go/Rust - fresh web service where latency and performance were more important than community/library support
• Haskell/Erlang - for very elegant/mathematical functional approach without a lot of business logic
• Clojure - in situation when you love Lisp (?)
• Kotlin/Scala - languages compiling to JVM bytecode (preferable over Clojure). Kotlin works with Java and has great IntelliJ support
• C - classes of applications (operating systems, language design, low-level programming and hardware)
• C++ - robotics, video games and high frequency trading where the performance gains from no garbage collection make it preferable to Java
• PHP/Hack - testing server changes without rebuilding. PHP is banned at Amazon due to security reasons, but its successor, Hack, runs a lot of Facebook and Slack's backends

Undefined nature of JavaScript can hide an error for a long time. For example,

function add(a,b) { return + (a + b) }
add(2,2)
add('2', 2)

will result in a number, but is it the same one?

Using PPS with Python

• Download ppscore: pip install ppscoreshell
• Calculate the PPS for a given pandas dataframe:

  import ppscore as pps
  pps.score(df, "feature_column", "target_column")
  

• Calculate the whole PPS matrix:

  pps.matrix(df)
  

Examples of projects that use documentation well

(chech the list below)

Class method should contain:

• brief description
• arguments
• label on default/optional arguments
• side effects description
• raised exceptions
• restrictions on when the method can be called

(check example below)

Type hinting introduced in Python 3.5 extends 4 rules of Jeff Atwood and comments the code itself, such as this example:

def hello_name(name: str) -> str:
    return(f"Hello {name}")

• user knows that the code expects input of type str
• the same about output

3 main categories of docstrings

Docstring conventions

Multi-line docstring example:

"""This is the summary line

This is the further elaboration of the docstring. Within this section,
you can elaborate further on details as appropriate for the situation.
Notice that the summary and the elaboration is separated by a blank new
line.

# Notice the blank line above. Code should continue on this line.

Example of using __doc:

Code (version 1):

def say_hello(name):
    print(f"Hello {name}, is it me you're looking for?")

say_hello.__doc__ = "A simple function that says hello... Richie style"

Code (alternative version):

def say_hello(name):
    """A simple function that says hello... Richie style"""
    print(f"Hello {name}, is it me you're looking for?")

Input:

>>> help(say_hello)

Returns:

Help on function say_hello in module __main__:

say_hello(name)
    A simple function that says hello... Richie style

init

Docstrings in scripts

You can always facilitate documentation with tools.

(check the table below)

Recommended videos to start documenting

(check the list below)

argparse

Different docstring formats:

• Google docstrings (not a formal specification)
• reStructured Text
• NumPy/SciPy docstrings
• Epytext

Public and Open Source Python projects should have the docs folder, and inside of it:

• Tutorials
• How-To Guides
• References
• Explanations

(check the table below for a summary)

dir() function examines directory of Python objects. For example dir(str).

Inside dir(str) you can find interesting property __doc__

Docstrings - built-in strings that help with documentation

help() function.

After typing help(str) it will return all the info about str object

project_root/
│
├── project/  # Project source code
├── docs/
├── README
├── HOW_TO_CONTRIBUTE
├── CODE_OF_CONDUCT
├── examples.py

(private, shared or open sourced)

Think only about """ when using docstrings

Python vs Fortran ;)

unittest Python library doesn't work well in a notebook

To unit test an exception handler:

worker = Worker(..., exception_handler=[handle_exception])
try:
    raise Exception()
except Exception:
    exc_info = sys.exc_info()

worker.handle_exception(job, *exc_info)

Although Miguel's tutorial is excellent, this companion piece by vsupalov can be helpful.

mkdocs for diagrams seems cool. :)

https://github.com/UnlockedEdu/documentation-pipeline-generator

How long to keep a failed job.

How long to keep a successful job.

You can add metadata on the job like keeping track of the number of times a job has been retried for example.

Exception handlers are attached to the worker.

Write an exception handler that requeues a failed job.

Debugging tests themselves

1. Set a breakpoint on the first line of the failing function (e.g. test_decrement)
2. Click the "Debug Test" option above the function
3. Open Debug Console and type: inc_dec.decrement(3) to see what is the actual output when we use x=3
4. Stop the debugger and correct the tests
5. Save the test file and run the tests again to look for a positive result

pytest-xdist provides support for parallel testing.

1. To enable it on Windows:

py -3 -m pip install pytest-xdist

1. Create a file pytest.ini in your project directory and specify in it the number of CPUs to be used (e.g. 4):

   [pytest]
   addopts=-n4
   

2. Run your tests

pytest displays failed tests also in PROBLEMS

Another way to view the test results:

View > Output > Python Test Log

Test Explorer is activated after discovering tests in Python:

After discovering tests, we can run them, for example, using CodeLens:

After using python.testing.autoTestDiscoverOnSaveEnabled, it'll be set to true and discovering tests whenever a test file is saved.

If discovery succeeds, the status bar shows Run Tests instead:

Tip to use when tests aren't discoverable in subfolderds (create __init__.py file)

Start testing in VS Code by using Python: Configure Tests (it automatically chooses one testing framework and disables the rest).

Otherwise, you can configure tests manually by setting only one of the following to True:

• python.testing.unittestEnabled
• python.testing.pytestEnabled
• python.testing.nosetestsEnabled

Default behaviour of test discovery by pytest framework

Default behaviour of test discovery by unittest framework

Sample test file using unittest framework. inc_dec is the file that's being tested:

import inc_dec    # The code to test
import unittest   # The test framework

class Test_TestIncrementDecrement(unittest.TestCase):
    def test_increment(self):
        self.assertEqual(inc_dec.increment(3), 4) # checks if the results is 4 when x = 3

    def test_decrement(self):
        self.assertEqual(inc_dec.decrement(3), 4)

if __name__ == '__main__':
    unittest.main()

Nose2 testing

• More on unit testing on Wikipedia
• useful examples on GitHub

e.g. test of an exact number entry:

    def test_validator_valid_string():
        # The exact assertion call depends on the framework as well
        assert(validate_account_number_format("1234567890"), true)

To run cells inside a Python script in VSCode, all you need to is to define Jupyter-like code cells within Python code using a # %% comment:

# %%
msg = "Hello World"
print(msg)

# %%
msg = "Hello again"
print(msg)

How JupyterLab differs from a traditional notebook

conda vs pip:

After creating a new project related environment, it shall be specified as a default for this specific project

Try to use logpoints instead of print statements.

More info: https://code.visualstudio.com/docs/editor/debugging#_logpoints

Command to quickly create a new Anaconda environment: conda create -n myenv python=3.7 pandas jupyter seaborn scikit-learn keras tensorflow

Enable history and sane keys in python shell

git-friendly Jupytext options include

• Julia: .jl
• Python: .py
• R: .R
• Markdown: .md
• RMarkdown: .Rmd
• and more!

The creator of Python, Guido van Rossum, did not intend for Python to have functional features but did appreciate some of the benefits its introduction has brought to the language

Python includes HOF to make processing iterable objects like lists and iterators much easier.

Immutability. While we can change the contents of a mutable object in a Tuple, we cannot change the reference to the mutable object that's stored in memory.

Exclamation in Python: $$x!$$ is written as:

x = 5
fact = 1
for i in range(x, 0, -1):
    fact = fact * i
print(fact)

it can be shortened as:

import math
math.factorial(x)

and the output is:

# 5*4*3*2*1
120

Hat in Python: $$\hat{x}$$ is written as:

x = [1, 2, 3]
length = math.sqrt(sum([e**2 for e in x]))
x_hat = [e/length for e in x]

This makes the magnitude of the vector 1 and only keeps the direction:

math.sqrt(sum([e**2 for e in x_hat]))
# 1.0

Dot Product in Python: $$X.Y$$ is written as:

X = [1, 2, 3]
Y = [4, 5, 6]
dot = sum([i*j for i, j in zip(X, Y)])
# 1*4 + 2*5 + 3*6
# 32

Element wise multiplication in Python: $$z=x\odot y$$ is written as:

x = [[1, 2], 
    [3, 4]]
y = [[2, 2], 
    [2, 2]]
z = np.multiply(x, y)

and results in:

[[2, 4]],
[[6, 8]]

Transpose in Python: $$X^T$$ is written as:

X = [[1, 2, 3], 
    [4, 5, 6]]
np.transpose(X)

and results in:

[[1, 4], 
 [2, 5],
 [3, 6]]

Function in Python: $$f:X \rightarrow Y$$ is written as:

def f(X):
    Y = ...
    return Y

Using R instead of X or Y means we're dealing with real numbers: $$f:R \rightarrow R$$ then, R^2 means we're dealing with d-dimensional vector of real numbers (in this case, example of d=2 is X = [1,2]

Epilson in Python: $$3 \epsilon X$$ is written as:

X = {1,2,3}
3 in X

Norm of vector in Python (it's like Pythagorean theorem): $$| x|$$ is written as:

x = [1, 2, 3]
math.sqrt(x[0]**2 + x[1]**2 + x[2]**2)

Absolute value in Python: $$|x|$$ is written as:

x = 10
y = -20
abs(x)  #10
abs(y) #20

PI in Python is the same as sigma, but you multiply (*) the numbers inside the for loop. $$\prod_{i=1}^Nx^i$$

Average in Python: $$\frac{1}{N}\sum_{i=1}^Nx_i$$ is written as:

x = [1, 2, 3, 4, 5]
result = 0
N = len(x)
for i in range(n):
 result = result + x[i]
average = result / N
print(average)

or it can be shortened:

x = [1, 2, 3, 4, 5]
result = sum(x) / len(x)

Sigma in Python: $$\sum_{i=1}^Nx_i$$ is written as:

x = [1, 2, 3, 4, 5]
result = 0
N = len(x)
for i in range(N):
 result = result + x[i]
print(result)

or it can be shortened:

x = [1, 2, 3, 4, 5]
result = sum(x)

2D vectors in Python: $$x_{ij}$$ are written as:

x = [ [10, 20, 30], [40, 50, 60] ]
i = 0
j = 1
print(x[i][j]) # 20

rdkit can measure dihedral. How to define the dihedral? seems user needs to provide 4 atom ids, but how to get the ids? Method 1. Use PyMOL to load the structure, and show label>Atom properties>Rank. Then the atom id will be labelled Method 2. Use SMARTS? How to?

with Python 3.7.4, the statement should be:

form['username'].value

use python to launch a http server

Python has a slight input into pharma, but it cannot outbeat SAS and R

I really don't understand Iterator. How can it be used to eliminate temporary lists?

Using Python scripts inside R Shiny (in 6 steps):

1. In ui.R create textOutput: textOutput("textOutput") (after plotoutput()).
2. In server.R create handler: output$textOutput <- renderText({ }].
3. Create python_ref.py and insert this code:
4. Import reticulate library: library(reticulate).
5. source_python() function will make Python available in R:
6. Make sure you've these files in your directory:

• app.R
• python_ref.py and that you've imported the reticulate package to R Environment and sourced the script inside your R code.

Hit run.

Vaex - library to manage as large datasets as your HDD, thanks to:

• memory mapping
• efficient out-of-core algorithms
• lazy evaluations.

All wrapped in a Pandas-like API

Keras's Tokenizer Class - Victor's preferred way of implementing BOW in Python

Keras always reserves 0 and never assigns any word to it; therefore, even when we have 6 words, we end up with the length of 7:

[0. 1. 1. 1. 0. 0. 0.]

Checking against None with == and !=

>>> x, y = 2, None
>>> x == None
False
>>> y == None
True
>>> x != None
True
>>> y != None
False

More Pythonic way by using is and is not:

>>> x is None
False
>>> y is None
True
>>> x is not None
True
>>> y is not None
False

Getters and Setters in Python:

>>> class C:
...     def get_x(self):
...         return self.__x
...     def set_x(self, value):
...         self.__x = value

Get and set the state of the object:

>>> c = C()
>>> c.set_x(2)
>>> c.get_x()
2

Iterating over Sequences and Mappings

>>> x = [1, 2, 4, 8, 16]
>>> for i in range(len(x)):
...     print(x[i])
... 
1
2
4
8
16

better way of iterating over a sequence:

>>> for item in x:
...     print(item)
... 
1
2
4
8
16

Reversing with indices:

>>> for i in range(len(x)):
...     print(i, x[i])
... 
0 1
1 2
2 4
3 8
4 16

Better way by using enumerate:

>>> for i, item in enumerate(x):
...     print(i, item)
... 
0 1
1 2
2 4
3 8
4 16

Iterating over a reversed order:

>>> for i in range(len(x)-1, -1, -1):
...     print(x[i])
... 
16
8
4
2
1

More elegant way:

>>> for item in x[::-1]:
...     print(item)
... 
16
8
4
2
1

Pythonic way of reversing an order:

>>> for item in reversed(x):
...     print(item)
... 
16
8
4
2
1

Defining some properties (considered to be more Pythonic):

>>> class C:
...     @property
...     def x(self):
...         return self.__x
...     @x.setter
...     def x(self, value):
...         self.__x = value

Result:

>>> c = C()
>>> c.x = 2
>>> c.x
2

Avoiding multiple optional arguments:

>>> def f(value, seq=[]):
...     seq.append(value)
...     return seq

If you don't provide seq, f() appends a value to an empty list and returns something like [value]:

>>> f(value=2)
[2]

Don't be fooled. This option isn't fine...

>>> f(value=4)
[2, 4]
>>> f(value=8)
[2, 4, 8]
>>> f(value=16)
[2, 4, 8, 16]

Iterating over a dictionary:

>>> z = {'a': 0, 'b': 1}
>>> for k in z:
... print(k, z[k])
... 
a 0
b 1

Applying method .items():

>>> for k, v in z.items():
...     print(k, v)
... 
a 0
b 1

You can also use the methods .keys() and .values()

The Zen of Python or PEP 20 <--- rules followed by Python

Unpacking <--- assign values

>>> a, b = 2, 'my-string'
>>> a
2
>>> b
'my-string'

Iterating over two or more sequences:

>>> y = 'abcde'
>>> for i in range(len(x)):
...     print(x[i], y[i])
... 
1 a
2 b
4 c
8 d
16 e

Better solution by applying zip:

>>> for item in zip(x, y):
...     print(item)
... 
(1, 'a')
(2, 'b')
(4, 'c')
(8, 'd')
(16, 'e')

Combining it with unpacking:

>>> for x_item, y_item in zip(x, y):
...     print(x_item, y_item)
... 
1 a
2 b
4 c
8 d
16 e

None (Python) ==similar== Null (C)

Chaining <--- checking if two or more operations are True

>>> x = 4
>>> x >= 2 and x <= 8
True

More compact (mathematical) form:

>>> 2 <= x <= 8
True
>>> 2 <= x <= 3
False

Unpacking <--- assign even more variables

>>> x = (1, 2, 4, 8, 16)
>>> a = x[0]
>>> b = x[1]
>>> c = x[2]
>>> d = x[3]
>>> e = x[4]
>>> a, b, c, d, e
(1, 2, 4, 8, 16)

more readable approach:

>>> a, b, c, d, e = x
>>> a, b, c, d, e
(1, 2, 4, 8, 16)

even cooler (* collects values not assigned to others):

>>> a, *y, e = x
>>> a, e, y
(1, 16, [2, 4, 8])

Unpacking <--- swap values

>>> a, b = b, a
>>> a
'my-string'
>>> b
2

Avoiding accessing protected class members Consider the code:

>>> class C:
...     def __init__(self, *args):
...         self.x, self._y, self.__z = args
... 
>>> c = C(1, 2, 4)

The instances of class C have three data members: .x, .y, and ._Cz (because z has two underscores, unlike y). If a member’s name begins with a double underscore (dunder), it becomes mangled, that is modified. That’s why you have ._Cz instead of ._z.

Now, it's quite OK to access/modify .x directly:

>>> c.x  # OK
1

You can also access ._y, from outside its instance, but it's considered a bad practice:

>>> c._y  # Possible, but a bad practice!
2

You can’t access .z because it’s mangled, but you can access or modify ._Cz:

>>> c.__z # Error!
Traceback (most recent call last):
File "", line 1, in 
AttributeError: 'C' object has no attribute '__z'
>>> c._C__z # Possible, but even worse!
4
>>>

**`with a`` block to handle exceptions:

>>> with open('filename.csv', 'w') as my_file:
...     # do something with `my_file`

Using the with block means that the special methods .enter() and .exit() are called, even in the cases of exceptions

Write beautiful Python code with

• Style Guide for Python Code
• PEP 8

PEP 8 provides the style guide for Python code, and PEP 20 represents the principles of Python language

Chained assignments <--- assign the same value to multiple variables:

>>> x = 2
>>> y = 2
>>> z = 2

More elegant way:

>>> x, y, z = 2, 2, 2

Chained assignments:

>>> x = y = z = 2
>>> x, y, z
(2, 2, 2)

Open a file:

>>> my_file = open('filename.csv', 'w')
>>> # do something with `my_file`

Close the file to properly manage memory:

>>> my_file = open('filename.csv', 'w')
>>> # do something with `my_file and`
>>> my_file.close()

__ <--- don't touch!

You can keep away from the problem:

>>> def f(value, seq=[]):
...     seq.append(value)
...     return seq

with some additional logic:

>>> def f(value, seq=None):
...     if seq is None:
...         seq = []
...     seq.append(value)
...     return seq

Shorter version:

>>> def f(value, seq=None):
...     if not seq:
...         seq = []
...     seq.append(value)
...     return seq

The result:

>>> f(value=2)
[2]
>>> f(value=4)
[4]
>>> f(value=8)
[8]
>>> f(value=16)
[16]

Comparing to zero - Pythonic way:

>>> bool(0)
False
>>> bool(-1), bool(1), bool(20), bool(28.4)
(True, True, True, True)

Using if item instead of if item != 0:

>>> for item in x:
...     if item:
...         print(item)
... 
1
2
3
4

You can also use if not item instead of if item == 0

Comparing to zero:

>>> x = (1, 2, 0, 3, 0, 4)
>>> for item in x:
...     if item != 0:
...         print(item)
... 
1
2
3
4

Use

• properties <--- when you can
• getters and setters <--- when you have to

Julia seems to be even faster than Scala when comparing to the speed of Python

Interesting estimation

Concurrent Python programs in 3 steps:

1. Write a script that carries out the task in a sequential fashion.
2. Transform the script so that it carries out the task using the map function.
3. Replace map with a neat function from the concurrent.futures module.

Fortunately, there is a workaround for concurrent programming in Python

Python isn't the best choice for concurrent programming

Replacing multithreading with multiprocessing:

replace ThreadPoolExecutor with ProcessPoolExecutor

Learn more about concurrency:

• Thinking about Concurrency + slides

Reason for multiprocessing being slower than multithreading:

Multiprocessing is more time-consuming to use because of its architecture

Faulthandler in contrast to tracing tracks specific events and has slightly better documentation

Most profiled parts of the software:

• Method or function (most common)
• Lines (similar to method profiling, but doing it line by line)
• Memory (memory usage)

Line profiling

cProfile > profile. Use profile only when cProfile isn't available

Memory usage can be tracked with pympler or objgraph libraries

Purpose of trace module

cProfile is one of the Python tools to measure method execution time. Specifically:

• number of calls (shown as ncalls)
• total time spent in the function (tottime)
• time per call (tottime/ncalls and shown as percall)
• cumulative time spent in a function (cumtime)
• cumulative time per call (quotient of cumtime over the number of primitive calls and shown as percall after cumtime)

Tracing (more for software devs) is very similar to event logging (more for system administrators)

We also want to measure the frequency of method calls. cProfile can highlight the number of function calls and how many of those are native calls

Basic Python libraries for tracing

Pandas usually writes code in this syntax:

1. JOIN
2. WHERE
3. GROUP BY
4. HAVING

Example:

1. df = thing1.join(thing2) # like a JOIN
2. df = df[df.created_at > 1000] # like a WHERE
3. df = df.groupby('something', num_yes = ('yes', 'sum')) # like a GROUP BY
4. df = df[df.num_yes > 2] # like a HAVING, filtering on the result of a GROUP BY
5. df = df[['num_yes', 'something1', 'something']] # pick the columns I want to display, like a SELECT
6. df.sort_values('sometthing', ascending=True)[:30] # ORDER BY and LIMIT
7. df[:30]

Nested for loop using list comprehension to come up with 5x5 matrix:

arr = [[i for i in range(5)] for j in range(5)]
arr
>>> [[0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4]]

List comprehension in Python to create a simple dictionary:

x = [2,45,21,45]
y = {i:v for i,v in enumerate(x)}
print(y)
>>> {0: 2, 1: 45, 2: 21, 3: 45}

Flattening a multi-dimensional matrix into a 1-D array using list comprehension:

x = [[0, 1, 2, 3, 4],
 [5, 6, 7, 8, 9]]

arr = [i for j in x for i in j]
print(arr)
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

2 examples of conditional statements in list comprehension:

arr = [i for i in range(10) if i % 2 == 0]
print(arr)
>>> [0, 2, 4, 6, 8]

and:

arr = ["Even" if i % 2 == 0 else "Odd" for i in range(10)]
print(arr)
>>> ['Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd']

pyenv allows you to easily switch between Python versions

__ methods

Dunder methods

Origins of Python name

Way of supporting a new field in JSON is much easier than in a relational database:

@property
def highlights(self) -> Sequence[Highlight]:
    default = [] # defensive to handle older export formats that had no annotations
    jsons = self.json.get('annotations', default)
    return list(map(Highlight, jsons))

Query language, not always = database. For example, see pandas

cachew tool isolates the complexity of database access patterns in a Python library

global inside a function is:

• required if we want to assign/change variable
• not required if we just want to print/access a variable.

Example code:

# This function modifies global variable 's' 
def f(): 
    global s 
    print s 
    s = "Look for Geeksforgeeks Python Section"
    print s  

# Global Scope 
s = "Python is great!" 
f() 
print s

nonlocal is used to declare variables inside the nested functions.

Example (if nonlocal wouldn't be used, the output would be "John", not "hello"):

def myfunc1():
  x = "John"
  def myfunc2():
    nonlocal x
    x = "hello"
  myfunc2()
  return x

print(myfunc1())

How to reduce Python expressions using the walrus operator (:=). It's used to both assign and evaluate variable names in a single expression, thus reducing repetition.

Before:

count = fresh_fruit.get('lemon', 0)
if count:
...

After:

if count := fresh_fruit.get('lemon', 0):
...

¿Cómo importar el Matcher de Spacy?

Spacy: atributos con y sin , como por ejemplo pos_ y id