how is this different to bi-directional RNNs?

this is a transfer because the CoVe can be trained on a task-specific dataset

i guess this is more sophisticated than the context count that we saw right at the beginning with word embeddings

the figure below is useful

embeddings are built by changing the embedding to increase the dot product between that word and its surrounding words. in this sense isn't it sort of like a special case of a language model? in that the model creates a probability distribution over sequences of words, i.e. how close two words are to each other in meaning

i.e. word embedding is a particular model that maps a word to a vector.

another example is a language model that, for a sequence of words, predicts the next token

entity extraction is another example of a pretrained model!

??

don't get this

neural machine translation

i.e. see whether a model can identify the meaning of these representations

why is that, if they get pruned anyway?

think about this more

what does that mean exactly?

wassat

if it's deterministic, how could it have different segmentations?

what for?

is this just ensure all vectors have same magnitude?

what exactly does vector normalisation mean here

don't get this

this is showing you what source token is heavily weighted when finding a given output

i.e. \(c^{t}\) is the context, which depends on the attention score

is the decoder state received the state from \(t-1\)?

again, how is it possible that it decreases the quality?

maybe because meaning decays on that length scale?

what does that mean? how is it possible?

don't really get that

for a specific task, requires additional data

in this case are the K top tokens all equally likely to be picked?

maybe change temp as you go through sentence? seems like you might want to seed more variation early on, then becomes important to increase coherence

are sentences fed a prefix? I guess if you're just sampling from a distribution, you can get different sentences with no input.

meaning? that the single layer has 1024 neurons? need to look into LSTMs

surely a dof is needed to allow the network to downweight these connections... Then again, maybe it's okay because there are gates that these combinations pass through later

not sure I fully get that

which is the same as maximising the log likelihood of \((y_1,y_2...,y_n\)

that is, we're trying to maximise the probability assigned to \(y_t\) – the correct next token/class given the context

I think the second equality below holds by definition of \(p_{y_t}\)

this connects with the Word Embeddings lecture. we're basically finding the cosine similarity between the input and output embeddings

and note that output embeddings have same dimension as input embedding of the context... does it make sense that context (which is generally composed of many words) has the same dimensionality as tokens (typically single words)

how do we know that this softmax calculation of probability is sensible? probably need to look at a general ML lecture for this

by training, this linear layer ensures that the V classes correspond to elements in the vocabulary. do we specify each class as being a particular element in the vocabulary?

i.e. next vocabulary token in the sequence

are these constant across the language model? or do they somehow vary based on context

i.e. almost 'just frequency based'

don't really get the difference between these two options

i.e. a way of determining the probability of a word given context

why?

otherwise overfit?

how do you ensure that a different feature is extracted each time?

don't know what this means

go back over probability course to remind what an MLE is

but it seems like there's a degree of freedom in this, i.e. what you need is the vector representation of the input text together with the vector representations of the classes, w, to give the right probabilities as an output

is the thing that makes it logistic, the softmax curve?

need softmax because if any probability goes to zero, you can never get it bak

just a non-square matrix multiplication, I think

w comes in below in the calculation of the probabilities, as defined above

i.e. pick parameters such that they maximise the joint probability?

e.g. if it was BOW, \(f_i\) would be the number of occurrences of word i

why?

i.e. we just used bayes' rule

presumably you could also use embeddings of words as features and it might perform better

think about why this is true, i.e. why just learning the conditional probability is equivalent to just finding boundaries between classes

relevant to NER and entity classification

interesting (related to disambiguating entities). an obvious approach is to look at the nearest neighbours of identical words within the corpora and see how similar they are – either simply count based, i.e. how many of the nearest neighbours are the same, or cosine similarity, e.g. what is the average cosine similarity of the nearest neighbours of the word

interesting to combine two different window sizes in some way

at a glance, it looks like it would be the same except for a non-linearity in the sum for \(u_w\) when w is the central word. come back to this

why called this?

come back to this

why is sigmoid used here but not when not negative sampling?

what does \(w\in V_{oc}\) mean below? and why does it disappear in fig 4 below?

presumably, the updates should be greater each time to compensate for the fact that each word is updated fewer times

why neg samples? because they're being downweighted?

why?

I thought Lena said above that the learned parameters are just the word vectors?

minimising the loss function is the same as maximising the likelihood. I.e. find the parameters \(\theta\) that will maximise the likelihood of the observation

how are these probabilities computed? clearly, that will determine what the suitable vectors are.

?

is this the matrix we saw above? terms on one axis, contexts on the other

how is document defined? is it a sentence?

?

presumably, this V is very big, right? e.g. 100000s of words. how do you determine what the tokens should be? i.e. what the domain of the output is

what exactly is this?

?

come back to this

seems analogous to the variance

the problem is that this is in practice inaccessible!

we need to choose a function on the sample that will estimate yield some desired characteristic of the population

presumably, this follows from the independence of the random variables (i.e. it's just the product of all the probabilities, with an \(n!\) to account for indistinguishability,

i.e. we don't care whether it's the first sample that's the smallest or the 15th, we only care what the size of the 15th largest value is

the probability that the \(i^{th}\) value (from smallest to largest) has a value x

random variable that gives the smallest value in a sample of n items

transforming the variable X to standard normal

follow from the above?

if you take an arbitrarily large sample, the probability of discrepancy between the sample mean and the true mean µ gets arbitrarily small

Think about the proof for this

i.e. it's a function of X

n is the number of samples used

don't 3 and 4 follow from 2?

what does uniformly and independently really mean?

this seems like a bad definition. What does it mean for a quantity to be random? The bit that is sent is not truly random. They seem to be different ways of conceiving of randomness.

i.e. there's noise at X in addition to the randomness of \(\Theta\)

i.e. the outcome of the presidential election/percentage of people that will vote for A is undetermined, i.e. it is itself a random variable with some distribution

is Y our random sample?

for example: the outcome of the presidential election. Get some polling data. From this data, estimate the most likely outcome.

i.e. what is the truth/true state, given some data about the truth that imperfectly (randomly) reflects the true state

seems like randomness is any source of discrepancy between the 'truth' of something and the record of 'the truth' that we receive

what exactly does randomness mean in general?

skipped, again because not too worried about integrals at this stage

skipped

tricky

Gamma is to Exponential as Pascal is to Geometric (which makes sense, since we said that exponential is the continuous limit of Geometric)

note that this follows from the general equation for variance transformation

actually, it doesn't follow that these distributions are necessarily normal

follows from linearity of expectation

probability that X>x, i.e. goes to zero for large x as expected

if F(x) = 1/2, does that make x the mode or median or mean? – median!

I've never understood exactly what that means

i.e. \(\int P(x) dx = 1\)

read this – chapter 7.1.2

thus \(\lambda\) is the probability of success per unit time

the bigger lambda, the more sharply peaked the distribution -> the smaller the variance

doesn't grow arbitrarily large at negative x

error

i.e. if g(x)is steep at x_i, then the function spends 'less time' at x_i, so it makes sense that the PDF should be reduced.

not just because it is a finite point, but also because it's a stationary point

for instance, in the graph above, consider the y-intercept to be 3.

Then there are independent solutions for g(x) = 3 in each of the 3 partitions – a, b, c from left to right – so that \(f_Y (y) = \) the sum of each of those 3 probabilities.

didn't fully get; worth looking over if it proves useful

where \(y = g(x_1)\)

this seems a dodgy proof; why are we differentiating wrt to the variable that parametrises our function

unique point by monotonicity

the region of X that corresponds to the above region of Y

the region of Y that's less than y

this is not the case for x^2 for instance, that's why sqrt(x) isn't well defined everywhere

pdf of X at x_1 / derivative of g(x) wrt x at x_1 at an x_1 where y = g(x_1)

surely this isn't generally satisfied, e.g. for a uniform distribution over some interval, the cdf is not strictly increasing over all space

why is this generally true?

why not?

the probability that event A happens is the integral of the regions of X that correspond to A happening

meaning?

is this still true if the PDF is only defined in certain regions of the real line?

not super happy with this one

but if the marble chosen is blue, doesn't this change the probability of picking a blue marble?

PMFs of DISTRIBUTIONS

x is the random variable in question.

• Bernoulli(p): Indicator variable, which is 1 if the outcome occurs (e.g. heads) and 0 if not.

• Geometric(x, p) = probability that you need to repeat the Bernoulli trial x times to get a success

• Binomial(x, n, p) = probability that in n Bernoulli trials you get x successes.

• Pascal(x, m, p) = probability that you need to repeat a Bernoulli trial x times to get m successes. This is a generalisation of the Geometric dist., where Pascal(x,1,p) = Geometric(x,p)

• Hypergeometric(x,b,r) = the probability that you get x blue balls in a sample of k red and blue balls.

• Poisson(x,L) = the probability that you get x events when you would normally get L.

is it kosher to split up the limits like that? i.e. lim(ab) ?= lim(a)lim(b)?

ok because k=0 and k=1 and k=2 etc. are mutually exclusive, so their intersection is just the sum of the probabilities of their occurence

why can't you renormalise the distribution by just saying it's the same as getting at least one email every 10/3 minutes?

probability that X (number of customers that enter the shop) = k in the time when, on average, it should be λ

most likely value

why is the value below always equal in probability to the mean (is that true?)

why exactly

probability that I throw the coin k times, if I want to observe m heads

k is number of times you throw the coin, m is the number of times you see heads that you want to know the probability of, p is the probability of heads

binomal

note that if X~P(v) and Y~Q(v), X+Y ~! P(v)+Q(v) in general.

Q: Is there a way to define a geometric dist if the underlying trials can have more than 2 outcomes? E.g. would you use this if throwing a die, for the number of times you had to throw until you got a 5?

so each event would have to have its own indicator random variable; if S = {a,b,c,d} then X_i = {1 if i, 0 otherwise}, for i in S.

for instance, if your random experiment is throwing a coin once, then S is {H, T} and X:S->R is {0,1}.

Question: does the notation f:S->R mean that each element in S maps to one element in R? I think yes: but in general, you may have multiple elements mapping to the same values in the co-domain.

for instance, I wrote down the distribution P(y) = p(1-p)^y for the probability of throwing a coin y times before getting a head, when the probability of getting a head is p.

nothing disappears to get the nicer form of the variance, it's just that the E[X]^2 terms subtract

intuitive: to find the probability that Y = y, add up the probabilities of all the x's.

pretty sure this is also true for normal probabilities, why is it being stated explicitly here?

why?

just gives the probabilities of each of the countable outcomes for X

Essentially, a RV lets us take the things in the real world (events, such as a sequence of heads or tails) and turn it into a new thing that behaves in the same way, i.e. the output of the function X:S->RealNos can be thought of kind of as a new sample space, and we can assign probabilities to it.

chronological/scheduled jobs

site adminsitration

given the additional info that C happened, we can no longer treat A and B as independent. I.e. the additional info that C leads to a dependence between A and B.

Just because some events are conditionally independent, doesn't mean they're unconditionally independent, and v.v. This makes sense because in fact there are always conditions on events, if only implicitly, so if conditional indpendence doesn't imply unconditional independence, the converse shouldn't hold either.

common sense generalisation of P(A|B) = P(A) for A independent of B.

probability that A n B n C is just the (probability that C happened given that AnB happened) x P(AnB) happened

P(A,B)

the key idea is basically that the sample set is reduced to C

why not just have Range(f) = codomain(f)

paused here