“secondary pathway” that takes the sequence of (integer) positions for the tokens, and from these integers creates another embedding vector

The input is a vector of n tokens (represented as in the previous section by integers from 1 to about 50,000).

Cluster-Based Over Sampling

EDx(z|A)[1f(x)=f(z)]≥τ,A(x)=1

The x-axis shows the feature effect: The weight times the actual feature value.

You can subtract the lower-order effects in a partial dependence plot to get the pure main or second-order effects

Well, that sounds stupid. Derivation and integration usually cancel each other out, like first subtracting, then adding the same number. Why does it make sense here? The derivative (or interval difference) isolates the effect of the feature of interest and blocks the effect of correlated features.

ALE plots are a faster and unbiased alternative to partial dependence plots (PDPs).

For each of the categories, we get a PDP estimate by forcing all data instances to have the same category. For example, if we look at the bike rental dataset and are interested in the partial dependence plot for the season, we get 4 numbers, one for each season. To compute the value for "summer", we replace the season of all data instances with "summer" and average the predictions.

An assumption of the PDP is that the features in C are not correlated with the features in S. If this assumption is violated, the averages calculated for the partial dependence plot will include data points that are very unlikely or even impossible (see disadvantages).

f(x)=^f(xS,xC)=g(xS)+h(xC)

innervation ratio

A tree with a depth of three requires a maximum of three features and split points to create the explanation for the prediction of an individual instance.

∑j=1feat.contrib(j,x)

Feature importance

Logistic regression can suffer from complete separation. If there is a feature that would perfectly separate the two classes, the logistic regression model can no longer be trained. This is because the weight for that feature would not converge, because the optimal weight would be infinite. This is really a bit unfortunate, because such a feature is really useful. But you do not need machine learning if you have a simple rule that separates both classes. The problem of complete separation can be solved by introducing penalization of the weights or defining a prior probability distribution of weights.

But usually you do not deal with the odds and interpret the weights only as the odds ratios. Because for actually calculating the odds you would need to set a value for each feature, which only makes sense if you want to look at one specific instance of your dataset.

days_since_2011 4.9 0.2 28.5

R2=1−(1−R2)n−1n−p−1

2.4 Evaluation of Interpretability

Application level evaluation (real task)

This article discusses interrupts on the Arduino Uno (Atmega328) and similar processors, using the Arduino IDE. The concepts however are very general. The code examples provided should compile on the Arduino IDE (Integrated Development Environment).

It’s very hard to get consciousness out of non-consciousness

low firing rates,

fenvi=fmin+⎛⎝1−UthMUifmax−fmin⎞⎠∗U

model is an assembly of phenomenological models,

All of the muscle fibers in all of the motor units of a given muscle tend to move together, experiencing the same sarcomere lengths and velocities

effects of crossover and dropout

Randomized clinical trials analyzed by the intention-to-treat (ITT) approach provide unbiased comparisons among the treatment groups

Additionally, one of my all time favorite papers (Fröhlich & McCormick, 2010) showed that an applied external field can entrain spiking in ferret cortical slice, parametrically to the oscillatory field frequency.

But spikes do not compute! The cells “compute”, dendrites “compute”, the axon hillock “computes”. In that sense, spikes are epiphenomenal: they are the secondary consequences of dendritic computation, of which you can fully infer by knowing the incoming synaptic inputs and biophysical properties of the neuron.

estimators is the prior covariance ΣΣφφ

If the sensor noise has an independent identical distribution (IID) across channels, the covariance of the sensor noise in the referenced data will be Σεεrεεr=σ2TrTTr

When the number of references drops to zero, the object deletes itself. The last part is worth repeating: The object deletes itself

CALLBACK is the calling convention for the functio

Calculus on Computational Graphs: Backpropagation

One benefit of SGD is that it's computationally a whole lot faster. Large datasets often can't be held in RAM, which makes vectorization much less efficient. Rather, each sample or batch of samples must be loaded, worked with, the results stored, and so on. Minibatch SGD, on the other hand, is usually intentionally made small enough to be computationally tractable. Usually, this computational advantage is leveraged by performing many more iterations of SGD, making many more steps than conventional batch gradient descent. This usually results in a model that is very close to that which would be found via batch gradient descent, or better.

And so it makes most sense to regard epoch 280 as the point beyond which overfitting is dominating learning in our neural network.

If we see that the accuracy on the test data is no longer improving, then we should stop training

It might be that accuracy on the test data and the training data both stop improving at the same time

test data

What is the limiting value for the output activations aLj

zLj=lnaLj+C