CONCLUSION - GPT's performance (percentile) compared to human test-takers was lower than reported by open ai - scaled score on the essay also deviated from "true" essay score => that could imply that the actual true percentile in UBE was lower than reported - reported score (298) is 28 higher than passing score => the essay scores would have to be extremely inaccurate to undermine the conclusion of Katz et al (that gpt passed the bar exam)

but still

RE-EXAMINING ESSAY SCORES 1. in regards to scaled MBE score computing methods were accurate with official methods 2. however in MPT + MEE (essays), it was significantly changed - to the point where you can question if others that blidnly graded it (according to official protocols), would give the same score = 3 key changes: 1. lack of use of a formal rubric (lack of grading guidelines like in NCBE, model answers etc) - katz et al. don't mention any rubric, only that they compared the answers to "good" answers from state of maryland - that's already problematic bc they didn't even specify what scores the "good" answers got, only that they passed - since it's unclear what scores they got & those answers were the basis for determining what score GPT received => it would make sense to assume GPT's answers' score should also be unclear 2. lack of NCBE training of the graders of the essays - Katz et al used a subset of the authors who were trained lawyers 3. blinding & consistent manner of grading - all graders have to first grade 30 "calibration" essays of variable quality to make sure consistent scores are assigned to similar quality answers - Katz et al. method didn't involve blinding like ^ - they had the authors give samples independent lawyers to grade & either "match" the grade given by authors or "exceed" that grade - but those lawyers as well didn't have NCBE training + all the issues above

ASSESSING EFFECT OF HYPERPARAMETERS

METHODS - above analysis found no effect of prompt on performance = possibly bc of lack of variety of the prompts used originally - to find out if prompts affected the performance at all, he tested 2 new conditions: 1. minimally tailored condition (compared to Katz et al.) - in terms of formatting & substance = 2. maximally tailored condition = highest performing prompt settings (like the original ones_ PLUS few-shot prompting => providing multiple example MBE questions with sample answers & explanations structured in a desired format

5 trials for each condition, the same temperature settings - 0.5 (bc previous study revealed it doesn't affect performance)

RESULTS - mean MBE accuracy throughout all trials: 79.5% in maximally tailored & 70.9% on minimally - scaled score in minimally woyld be approx. 150 => thus would place it on 70th percentile among july takers, 64th among first timers & 48th percentile among those who passed - maximally tailored condition score was 164 - 6 higher than in original papers => this would result in 95th percentile in july takers, 87th among first timers & 82th among those who passed

original prompts included these ^

modifying prompts in minimally tailored condition = none of these were used, only

Please answer the following question,” followed by the question and answer choices (a technique sometimes referred to as “basic prompting”:

REPLICATING MBE SCORE

METHODOLOGY

1. materials: official MBE questions released by NCBE
2. procedure: replicating scores from open ai by following protocol documented by Katz et al.
3. testing performance by using 3 different temperature settings (0, 0.5 & 1)
4. for each of those, performance was tested with 2 different prompts (for each gpt was told to answer as if it were taking the actual bar exam):
5. asked to provide top-3 ranking of possible answers + justification + citation for the answer
6. the same but without justification + citation
7. for each of those 3 different trials to control for variation

WHAT MARTINEZ ADDED - additional temperature settings: 0.25 & 0.7 - thus total prompts for temperatures was 10, not 6 - 5 trials instead of 3 => total trials was 50 instead of 18

after prompting, they calculated raw scores (using official answer keys) -> then scaled scores (multiplying the raw score by 190, then dividing by 200, then converting to a scaled score using official NCBE data)

RESULTS: -mean accuracy here: 75.6% (original: 75.7%) - GPT'S RAW ACCURACY WAS NOT SIGNIFICANTLY LOWER/HIGHER AT A GIVEN TEMPERATURE SETTING OR WHEN FED A CERTAIN PROMPT

(context) parameters = configurations that determine how the NN will manipulate data and make predictions for NNs, parameters are basically weights & biases - they simplify identification of machine learning data; they develop how a NN propels data forward (forward propagation) - once that's completed, the NN will refine all the connections based on the errors that occured during forward propagation -> that leads to backwards propagation - the flow going backwards through the layers & connections of NN to readjust

WEIGHTS - manage connections between two basic units = neurons in a NN (weights of those units' signals are decreased/increased to train those units to move forward in the NN, during forward propagation) BIASES -

(context) NEURAL NETWORK "an algorithm built to work like a human brain. It is composed of multiple layers of neurons. It starts with an input layer consisting of independent neurons that do not rely on any weighted signal. It introduces primary data. The input layer then feeds into one to two hidden layers. The hidden layers contain neurons and biases that place value to data that sorts everything into the output layer. The output layer expresses the data identification for machine learning models"

research questions/goals 1. assuming the 298 score is legit, does it warrant the 90th percentile claim? 2. is it worth questioning the 298 claim? (replication, reproducibility - can it even be verified?) 3. given various settings/parameters in GPT-4, is it worth assessing how adjusting those settings will affect GPT's performance?

methodology

• paper attempts to replicate the reported MBE score using methods as close as possible to original papers
• comparing MBE performance using various (best & worst) hyperparameter settings
• re-examining performance on the essays by

a) evaluating to which extent the methodology of grading GPT4 essays deviated from official protocol used by NCBE during actual bar exam

b) to which extent such deviations might undermine one's confidence in the scaled essay scoers reported by openai

so far he didn't question the scaled score = 298 reported by openai

RESULTS:

against 1st timers: - each component, as well as overall UBE score: percentile among july 1st timers LESS than of OpenAI estimate& july estimate that includes repeat takers - overall UBE: scored in 62nd percentile (instead of 90th estimate in february & 68th in july) - MBE: 79th percentile (instead of 95th & 86th) - MEE + MPT: 42nd (instead of 69th & 48th)

against attorneys - results dropped even more: - UBE: 45th percentile - MBE - 69th - MEE + MPT - 15th GPT 3.5 = 0pth percentile for all

basically the score that's a passing score, to filter out unqualified candidates

here it is a % of people who didn't pass

basically shows how results are scattered within the distriibution and how they differ from the mean

the bigger the SD the more the results vary

measures how many standard deviations above or below the mean a data point is

basically z score tells u the distance between a particular score and the mean (how far that score is from the mean in a distribution)

COMPUTING STANDARD DEVIATION FOR 1ST TIMERS SCORES:

MBE score 1. taking publicly available distribution of MBE scores available on NCBE website (july scores, not for 1st time scorers cuz this is unavailable) 2. assuming that first timers have approximately the same SD as population in july 3. computing the SD deviation of 1st timers taking MBE - entering that publicly available distribution of MBE scores into R (program) - taking that SD of this distribution & using the built in sd() function (which calculates SD of a normal distribution)

essay scores exactly the same (bc mean & SD is the same as MBE)

UBE SD aren't publicly known for official exam, but they can be concluded by combining mean UBE score for 1st timers (287.6) & 1st time pass rates

mean of overall bar exam scores

essay is the same as MBE = 143.8 mean

mean MBE score of 1st timers = 143.8

in statistics: population = a whole group of objects that's the focus of the study

sample - a certain number of those objects

parameter = a number reffering to & describing the entire population statistic = ~||~ only it reffers to the sample

METHODOLOGY: - compute parameters from UBE, from MBE & from essays - generate distributions with these parameters

COMPUTE PERCENTILES OF GPT'S SCORE - compute what % of values on these distributions are lower than GPT scores = to estimate percentile against 1st timers - compute what % of values above the passing threshold are lower than GPT score = to estimate the percentile against attorneys

1st step of methodology - using data from more accurate estimates (for GPT 3.5 and 4) instead of the estimates from Open Ai & the july estimate = for that, he estimated scores separately for MBE, for essays & the overall UBE score

how could the percentile estimation be made more accurate? - using only scores of people who take the bar exam for the first time - when it comes to comparisons with attorneys' results, the results should be limited to both 1st time takers AND those who achieved a passing score - the evidence they used is based purely on Illinois exam data (because there's no official ones) - which is not identical to the actual UBE scoring & content = more accurate would be to use data directly from official NCBE sources

another source of evidence - this one is well documented & transparent about its methodology BUT it doesn't focus on percentiles, like the original report - it focuses on the model's score compared to average test takers

example of lack of backing up the evidence

no proof for those claims

where the evidence is

structure of the paper: 1. evaluating the 90th percentile claim: 4 findings about the actual bar exam performance 2. investigation of validity of the score 3. investigation of adjusting temperature settings 4. conclusion: estimates of the percentile are over inflated

possible outcomes = for AI: misuse of the models, releasing them without proper assessment

possible consequences = for humans: use of AI by lawyers

2nd methodological uncertainty = no knowledge on how they evaluated gpt's performance

1st methodological uncertainty = lack of official percentiles

previous scores on the bar exam

a database of text that's used for training/ testing models SO - GPT can take the test "open-book", because it has this huge collection of texts, while humans don't

doubts about the BAR exam itself - in regards to testing human knowledge too

zero-shot - you give the model a command to generate a response, but you don't provide examples of solving the task few-shot - you provide a few examples, so that the model can learn to solve tasks that it wasn't programmed to do

(that setting controls randomness, creativity of generated text = lower temperature means more predictable, higher means more creative, diverse)

paper debunks gpt's score of 298 - shows methodological issues in grading the exams

4TH CLAIM performance against only those who passed the exam = gpt scores ~48th percentile, ~15th on essays

3RD CLAIM = gpt's performance against actual 1st time takers is ~62nd percentile, ~42 percentile on essays

SECOND CLAIM = data from more recent SAME exam suggests the score was actually BELOW the 69th percentile, 48th on essays

first claim - the score does near the 90th percentile BUT those findings lean towards those who repeat the test after failing in july = they score significantly lower than the general takers

goal of the paper = debunking OpenAI's claim 4 claims

uh-huh. because parent = mother and mother only

why is it increasingly antagonistic? isn't that the past? where christians would be murdered in ancient rome? jews, muslims. isn't it rather that people become skeptical, and thus THEMSELVES antagonistic? sure, it does project onto society, but i don't think it's oppressing

who says that? in what aspect? in estabilished laws or just geneally as a right? check

contractualism - a philosophical idea (moral philosophy) 1. BROAD SENSE: - morality comes from agreements between people - "what we owe to each other" - as people, we have duties towards others, since we're all rational beings 2. NARROW SENSE (T. M. Scanlon): - different from the broad sense, because (as Scanlon claims) they are PRACTICAL claims about what we have a reason to do, not just theoretical (furthermore, they're the most important, because when we decide that an action is wrong, gives as reasons why we DON'T do that action, especially comparing them with other reasons) - an action is morally bad if it cannot be explained as good to another human being

"because they didn't want the protest to be political - it was moral" - prof. Benson

sometimes the biggest harm is passiveness, indifference. because openly evil people exist, but it's the white moderate who makes the majority

definitions of just/unjust law

through discerning which laws are in harmony with "eternal/natural law" and you can do that, knowing that "any law that degrades human personality is unjust"

there are types of laws that recquire breaking them, they're unjust

purposes of nonviolent direct action

2 different kinds of liberalism 1. convergence/illiberalism - in the end liberalism will force their liberal/tolerance onto others 2. modus vivendi/tolerance - disagreeing but finding a way to live together

Mill said liberalism is freedom to do whatever you want, until you hurt someone else

you can never establish those things, those are the very cores of faith and religions; they're not proveable

anti-metaphysical claim - very significant in that time, because it influenced thinking about natural law

G. J. Holyoake (1851) defined secularism: modern dictionaries cite/rephrase his definitions, but in reality (Benson found) that in his works, he's not at all neutral, but rather anti-religion - basing off of Comte, Rousseau

definition in poland

definition in austria

definition in denmark

definition in france

components of religion

german criteria

portugese criteria

italian criteria

research????//

lack of definitions in state-level regulations

in the end, Moens (somewhat surprisingly, since he pointed out all the evidence and ways that the guards could be found guilty) concluded that the trials resulted in "injustices masquerading as justice", mainly because the court failed to consider that applying West Germany critical tradition (based more on natural law), to East Germany soldiers, which were only familiar with the German tradition (based on Lutheran idea of total obedience to the law) was UNJUST

all three courts found the guards guilty, but for different reasons

Radbruch's 5 minutes article

Radbruch's first minute

so, normally 315 of Reunification Treaty prevents acts commited on East German soil prior to reunification from being punished, if they were not punishable under East German Law (kinda like our warunek podwójnej karalaności)

BUT, here: - immunity does not apply where there was already West German Law (west german law applied to crimes on foreign soil if): 1. the acts are commited against German 2. the person that committed them becomes a resident of WG or comes to WG

prof. Samson argues that EG became part of WG, so their law is applicable 7(2) (similarly, the people who were shot, were Germans, so 7(1))

BUT it's a stretch, because EGs were considered foreigners by WG

the BGH was reluctant to invoke natural law, because it's hard to define what the "minimum content" should be. instead, the relied on the international human rights - but they found that border shooting on itself does not violate the Covenant, only its excessive nature or unnecessary use violates it

so basically, the critical tradition was imposed on the EG guards, which was unjust because this idea was foreign to them. they were still followin the German tradition, that requires total obedience to the law

critical tradition in East Germany = absent

critical tradition in West Germany

Aquinas on natural vs positive law

the german tradition (influenced by Luther)

different traditions of thinking; the critical post-war tradition was applied to the EG soldiers, who followed the German tradition to obey the law

internal morality = Fuller

different interpretations of justifications

!!!!!!!!!!!!!!!

2 possible justifications in EG law: 1. appealing to the fact that use of firearms is permittable if used against a serious crime (defined as carried out with dangerous means; court later found that ladders that were used to climb the Berlin Wall were considered 'dangerous means', thus use of firearms was necessary) 2. explicit statement that soldier who follow orders are not criminally responsible (unless a blatant violation)

APPLICABILITY OF WG LAW milder law is applicable UNLESS there was already WG law at the time of the act - territoriality principle = WG law applicable to act in EG, if the consequences occured in WG - act was committed against a German permanently resided in WG - act carried out by West German - perpetrator moved to WG before the reunification

like our article 4 paragraph 1 - lex mitior agit

PRINCIPLE IV, mainly - earlier called "superior orders" - now reffered to as the Nuremberg Defense

essence of erastianism

reasons for temporariness of theocracies 1. lack of secular skills and means to run modern economy - by religious leaders 2. unwillingness to entertain the compromises of political + international relations

for short lives of theocracies

essence of theocracy

applying economic models to religions

goal of the paper

ad FAIRNESS 2 things have to be provided, before you can call a law "based on the moral concept of fairness" (to dispute the previous two arguments):

1. laws have generally beneficial effects
2. most other people obey the law (so that if you don't you benefit unfairly)

COUNTER ARGUMENTS TO FORMER IDEAS:

ad. 3 - fairness - anarchist: denial of any benefit coming from the law - those who claim it's beneficial: fairness can't apply, where obeying the law does no one any good, is useless (eg. it's not unfair to speed over the limit on a deserted road)

ad 4. public good - act-utalitarianism: there are of course examples, where total obedience to law does more harm than good - eg. man who earns a small income from a service to a friend, he doesn't tax it - if he would, the public would benefit, but his family could starve to death

BUT generally law-breaking sets a bad example (for the kids, others etc); its consequences include imitation of further law-breaking - rule-utilitarianism = it's difficult to compare all different rules and their consequences

ALL FOUR CONCEPTS OF WHY YOU SHOULD FOLLOW THE LAW: 1. gratitude = your country & law was the source of great benefits for you, so you should at least obey the law but against, you could argue that you can be grateful to many people, but it doesn't mean you have to obey everything they say 2. promise-keeping: citizens promise to obey the law in exchange for protection & other benefits (kind of a "social contract" like in Rawls' theory) 3. fairness: different from promise-keeping, because it's extended to all citizens, as a moral ground to everyone, not just to those who choose to participate in the politics SO, you should obey the law, because it would be unfair not to; you owe your fellow citizens "if they all comply and you benefit, it is unfair if you benefit without complying" 4. public good = if people break the law, the welfare of society is diminished, thus we're all morally obliged to obey

different forms of the utalitarian concept: - act-utalitarianism = an act is morally wrong if it'll have worse consequences than other acts possible on this occasion - "rule-utilitarianism": you should obey the law, if it's required by a rule, that leads to best consequences when observed objectively

"rule-utilitarianism": you should obey the law, if it's required by a rule, that leads to best consequences when observed objectively

BUT it's difficult to compare all different rules and their consequences

most common justification of obeying the law: public good (there are of course examples, where total obedience to law does more harm than good - eg. man who earns a small income from a service to a friend, he doesn't tax it - if he would, the public would benefit, but his family could starve to death

counter argument to ^ = act-utalitarianism: not taxing the income, sets a bad example (for the kids, others etc); its consequences include imitation of further law-breaking

2 things have to be provided, before you can call a law "based on the moral concept of fairness" (to dispute the previous two arguments): 1. laws have generally beneficial effects 2. most other people obey the law (so that if you don't you benefit unfairly)

to the argument of fairness: 1. anarchist: denial of any benefit coming from the law 2. those who claim it's beneficial: fairness can't apply, where obeying the law does no one any good, is useless (eg. it's not unfair to speed over the limit on a deserted road)

duty to obey law comes from fairness: different from promise-keeping, because it's extended to all citizens, as a moral ground to everyone, not just to those who choose to participate in the politics SO, you should obey the law, because it would be unfair not to; you owe your fellow citizens "if they all comply and you benefit, it is unfair if you benefit without complying"

duty to obey the law comes from promise-keeping: citizens promise to obey the law in exchange for protection & other benefits (kind of a "social contract" like in Rawls' theory)

a good way of illustrating just how relative natural law (or the core principles) is. because at what point, to which extent and based on what, do we decide when to obey or disobey the law?