Epistemedia

How We Know · Case 003 · skeptical

What GPT-4's 90th-percentile bar-exam claim compared

How did a historical simulated UBE score reported for GPT-4 become a roughly 90th-percentile claim, and how does the rank change when the comparison population changes?

Historical evidence through 2026-08-27 about one simulated UBE score and the comparison populations used to rank it. This evidence file does not describe current model behavior, practicing-lawyer quality, or general legal competence.

skeptical finding

withhold general 90th-percentile and lawyer-quality claims

What to do with that: Do not repeat '90th percentile' without its missing comparison distribution, and do not translate the result into lawyer quality.

Official-chart sensitivities and the modeled re-analysis answer narrower questions and are not current product rankings.

Lineage accounting

The units count different things

1historical model runReport, paper, repository, and supplement share one experiment3official administration sensitivitiesThe same score ranks differently across official Illinois charts7rooted evidence unitsSeven units after editions and reused score inputs are collapsed3unresolved boundariesVisible limits receive no invented value or broader interpretation

Shared capture: The report, research paper, repository, and supplements are manifestations of one historical model run; the Martinez analysis reuses that score and aggregate comparison data.

Warrant boundary: Percentile is not a property of the score alone. It depends on the administration, population, cutoff, interpolation, and model assumptions.

Sentence x-ray

What the selected record actually supports

Open a sentence to inspect its exact work, edition, span, retrieval, digest, and license chain.

01 The launch report says test takers but does not disclose a UBE administration, jurisdiction, population composition, chart, or interpolation for the displayed percentile.

Typed relation: support

Relation
relation-assertion-claim-launch-comparison-unspecified em:dossier-evidence-relation:sha256:affd0ea72f66e971c487ac561d1d99587c411a4aade9a9e4908ea32f842f20c5
Proposition
claim-launch-comparison-unspecified em:dossier-proposition:sha256:0e95785a0aeb86c6b85771df4f9717da9521e2f9e5d52d608e451465e1240d2f
Assertion
assertion-claim-launch-comparison-unspecified em:dossier-assertion:sha256:e5b09f98417934e590aa916b7e5c657dc707fb55ab33c81d87a6a724d48fc159
Lineage
lineage-model-performance-root em:dossier-lineage:sha256:c8b81e09b0245e5c5302dd89be2d627f7658d7ab0d5de6d56adef3df0f65e145

Typed dependence

  1. edge-author-social-collaborationlineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:b616095462050178a4dedd71e5ad524effaa71d20808eceb206d203f9aae8204accepted_edge_id=edge-author-social-collaboration; accepted_dimension=author-social; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=The vendor report and study are socially linked manifestations, not independent tests.
  2. edge-citation-katz-to-martinezlineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:a8c16efd90634c0a152f0fdd7a0a02f8d844004ea05edec24ec4651bf00764ccaccepted_edge_id=edge-citation-katz-to-martinez; accepted_dimension=citation; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=Citation links the analysis lineage to the same reported score.
  3. edge-data-reported-score-reuselineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:691d48bc190063d50b6e811c7455ec325b1f29bcc7452301694470e83211188aaccepted_edge_id=edge-data-reported-score-reuse; accepted_dimension=data; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=The re-analysis is not a new model-performance experiment. | Score validation changes warrant, not the participant or model-performance root count.
  4. edge-material-shared-exam-itemslineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:8f61eda50599e5c6c226c2dcb7818c8627f23979fd5b51669aa62e5d52f6d110accepted_edge_id=edge-material-shared-exam-items; accepted_dimension=material; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=All score manifestations inherit the same exam-item materials.
  5. edge-method-single-free-response-runlineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:1ed5b88123954bc8da25cf9422c2df70e30d3569ddd99213e9cf5590bf8522c3accepted_edge_id=edge-method-single-free-response-run; accepted_dimension=method; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=Repeated documents do not supply repeated-run independence.
  6. edge-model-historical-snapshotslineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:e7c0af1eebfe5fa400bcb3369655f40adf482a99cc91a370f6e6a586f847f133accepted_edge_id=edge-model-historical-snapshots; accepted_dimension=model; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=The two snapshots are components of one reported experiment, not two evidence roots.
  7. edge-score-component-compositelineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:21278e42f97e114bc1442703ca96eea56725d7bb221a07e5c31f5adfc512b040accepted_edge_id=edge-score-component-composite; accepted_dimension=score; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=297 and 298 are scoring variants within one experiment.

Appendix for GPT-4 passes the Bar Exam

footnote 5

there is no publicly available July 2022 national bar exam percentiles against which to compare these results
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:ca6ba91cbc68c2a3aa343fc059aa3058d27baa364e998d9c05393578c6045d5c
Span digest
sha256:53c5c3cd65f0c06ae12ec339f9a8f503b977ebaf9c1cd8723d9caecfa42bcdcb
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

GPT-4 Technical Report

Appendix A.5, PDF p. 25

Percentiles are based on the most recently available score distributions for test-takers of each exam type.
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:2400a4ddb632abda72527df1b7e8f30dd7f400818992926a8e8119c057e9aff5
Span digest
sha256:5abbca6a7e803506a3ed9db112873db65bfb4ba80dde648b0ba7b9ae96161a9f
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

GPT-4 Technical Report

Appendix A.1, PDF p. 23

The Uniform Bar Exam was run by our collaborators at CaseText and Stanford CodeX.
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:61f2be1da866e5764042a9022c4a0a0eb5d320a0df71f95a24a1a1eb5fd271ac
Span digest
sha256:e0b72dc61e4614a425e991b3fd9641db637b3516311659728f615bed0ea4100c
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

Appendix for GPT-4 passes the Bar Exam

footnote 5

While we are not fully convinced of the methodological approach taken in some subsequent analysis [78], we do agree that it would be better to consider the raw 297 UBE as falling within a range between 68th and 90th percentile (depending on the precise state and timing of the exam administration).
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:477c595f89b6efc4f4fac04d627e65d95daa7929d10f76d3486a96c6c8bf8b06
Span digest
sha256:7e376175b36f8e2809bc74e378c7354858289a689370b70f82df2965d32b9026
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

Appendix for GPT-4 passes the Bar Exam

§4(d), Table 7

{ "cells": [ { "cell_id": "cell-katz-gpt4-mbe", "column": "GPT-4", "row": "MBE", "text": "157 points" }, { "cell_id": "cell-katz-gpt4-mee", "column": "GPT-4", "row": "MEE", "text": "84 points" }, { "cell_id": "cell-katz-gpt4-mpt", "column": "GPT-4", "row": "MPT", "text": "56 points" }, { "cell_id": "cell-katz-gpt4-overall", "column": "GPT-4", "row": "overall score", "text": "297 points" } ], "format": "table-cell-transcription", "locator": "\u00a74(d), Table 7", "normalization": "exact-cell-text", "span_id": "span-katz-vor-components", "supports": "The version-of-record component scores sum to 297." }
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:293687df5cf4b74d839237414173205591aacf4850103e1621aece4f21fb763b
Span digest
sha256:e6c808d4b4dcd243210518497a0c8788d06ee91d8571c474c13514c2c8afe04b
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

percentile_analysis_new.Rmd

introduction, journal p. 584

The paper successfully replicates the MBE score of 158, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the essay score (140).
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:47ce609849ae1ff1224df62ce9b453707dd9b7a93cd4413822c84178eac7f853
Span digest
sha256:11db34f6066f861f338266e98c11852743b727734f62c1a773ae257bf219e609
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

Appendix for GPT-4 passes the Bar Exam

§3(a), materials

{ "format": "exact-segments", "locator": "\u00a73(a), materials", "normalization": "collapse-whitespace", "segments": [ { "segment_id": "segment-katz-mee-mpt-materials", "text": "For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination." }, { "segment_id": "segment-katz-mbe-materials", "text": "The MBE questions used in this study are official multistate bar examination questions from previous administrations of the UBE [65]." } ], "span_id": "span-katz-vor-materials", "supports": "The study components share disclosed exam-item materials and administration lineage." }
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:076f2679614413d88a366ceec578a4c96672df24e2fe9dc8afe1242c8c7a5496
Span digest
sha256:9107c310f1bb38d087940d7432c989d2ab81d8e79356bc6a42a6c64bbf2db37e
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

GPT-4 Technical Report

Appendix A.3, PDF p. 24

we simply ran these free response questions each only a single time at our best-guess temperature (0.6) and prompt
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:8b5e9985c61e0fea14503003ec6422ec54340f4069def5d98d5e1b7147da6ba6
Span digest
sha256:eebbf2c0c2999d72a017e9af6b4799d65577fb0715ac5d85e6ee757e05aae78e
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

GPT-4 Technical Report

Appendix A.6, PDF p. 25

We ran GPT-4 multiple-choice questions using a model snapshot from March 1, 2023, whereas the free-response questions were run and scored using a non-final model snapshot from February 23, 2023.
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:6f6629457faeeb750d379bc919a2030743b0fd07db285521e3a54a46f02ab3ab
Span digest
sha256:9dc5f4260c333122ddb37e7ca7e2f3eecd1d1976475145759b772a7701c58f1e
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

Appendix for GPT-4 passes the Bar Exam

footnote 4

Best prompt and/or hyperparameter combination on the MBE would push this score to 298 or higher. Here, we report the MBE average of 75.7% which composites to a 297.
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:5dd561ae54601a051ef573af4ec859665f4f1d2fb801788a1339de8b16dd44fc
Span digest
sha256:0ce9516e5b95f3b35121d471b8ab8d39b67199ffb40941deb28e07e2a07926ab
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree
02 The same disclosed linear interpolation places 298 near 67.8th in the July 2018 Illinois chart.

Typed relation: support

Relation
relation-assertion-claim-july-sensitive em:dossier-evidence-relation:sha256:73716abf204fa15d841ff6b95c2dfbe303e3fcb4e06c26923a8fa5a618593415
Proposition
claim-july-sensitive em:dossier-proposition:sha256:c03843891cd010c53ddbe03e8d0128baeeccd8dedde381d2aaf3ed9a8bf08752
Assertion
assertion-claim-july-sensitive em:dossier-assertion:sha256:8feb34c96f4f746c4cea970e4ced3e891be6c2faa4d74e8370abed8ab69e8371
Lineage
lineage-illinois-comparison-root em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20

Typed dependence

  1. edge-benchmark-illinois-chartslineage-illinois-comparison-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:3bbbc190e0662f0d0cf22c4bf02a5abe6f9e71e95061afd3dc94f535c83fb57aaccepted_edge_id=edge-benchmark-illinois-charts; accepted_dimension=benchmark; from=lineage-illinois-comparison-root; to=lineage-martinez-analysis-root; effects=It is comparison data, not an additional GPT-4 performance root. | Population drift affects rank without changing the model score. | It is a sensitivity input, not a replication of the model run.
Calculation · reviewer sensitivity only: linear interpolation: 89.0; equation="p298 = p290 + ((298 - 290) / 10) * (p300 - p290)"; comparison_population=null; depends_on=[].
Relation
relation-assertion-derive-illinois-feb-2018-298 em:dossier-evidence-relation:sha256:89e465da9ccc872747e2d6c2cf69dbffd57f4c4e3ddd1ec6974b3ce4f3653ecc
Proposition
em:dossier-proposition:sha256:b3eb260778bfb82eeb304b0af6d0f95ae5970cb1e83f64f824eadc453cf0a11f
Assertion
em:dossier-assertion:sha256:f5102c2dfb6527bc325f323f122a10c78a78bc7422ed053fbe00e8edf7d6ab1d
{
  "derivation": {
    "derivation_id": "derive-illinois-feb-2018-298",
    "equation": "p298 = p290 + ((298 - 290) / 10) * (p300 - p290)",
    "input_cell_ids": [
      "cell-illinois-feb-2018-290",
      "cell-illinois-feb-2018-300"
    ],
    "input_span_ids": [
      "span-illinois-feb-2018-anchors"
    ],
    "inputs": {
      "p290": 85.0,
      "p300": 90.0,
      "score": 298
    },
    "method": "reviewer sensitivity only: linear interpolation",
    "result_percentile": 89.0,
    "uncertainty": "Neither Illinois nor OpenAI disclosed this interpolation; it cannot be attributed as the launch method."
  },
  "resolved_input_cells": [
    {
      "cell_id": "cell-illinois-feb-2018-290",
      "percentile": 85,
      "score": 290
    },
    {
      "cell_id": "cell-illinois-feb-2018-300",
      "percentile": 90,
      "score": 300
    }
  ]
}
  1. span-calculation-derive-illinois-feb-2018-298accepted derivation and input cells: derive-illinois-feb-2018-298em:dossier-span:sha256:ba2cde38323123970a17e90771d1f6e21ad66d1ddfa53841749b93e1113694ea · sha256:3037e78d052c9ae7ef8fc027de340daa7e7a1e7488092ce8c1d4ab46cfe1762eem:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-illinois-feb-2018-anchorstotal-scale table, rows 300 and 290em:dossier-span:sha256:56a011a09e4e7dd05afbc0309c94d4dcfa1ecf1d6fd3df6a6a6f65204c7ccf03 · sha256:b3b8f789db7571a4fd8ed73e5380f2847aae920172722b9f6f51eec96c62e9ceem:dossier-edition:sha256:9be0e724fc35f74c96e635c6263f8f44ef804c3406c34b268bb710b24228a368 · sha256:44de7bd89cd4aeacee7b38d7077ff659e3baf552cb9b295401ec9b7229c04964em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed · no open license confirmed
Calculation · reviewer sensitivity only: linear interpolation: 88.6; equation="p298 = p290 + ((298 - 290) / 10) * (p300 - p290)"; comparison_population=null; depends_on=[].
Relation
relation-assertion-derive-illinois-feb-2019-298 em:dossier-evidence-relation:sha256:eab848ea50c1f7403f46775de29a61ab145b1ef616b6ddedd1befaf47c47d626
Proposition
em:dossier-proposition:sha256:5e417788588abd558e96b1e86a8abbf760107f1b2c5cf87ab39ef6533de01c6f
Assertion
em:dossier-assertion:sha256:2c467a78217c9cfd8e1927f656d2c404085fc79f1544227ad10737e3c0e96bb4
{
  "derivation": {
    "derivation_id": "derive-illinois-feb-2019-298",
    "equation": "p298 = p290 + ((298 - 290) / 10) * (p300 - p290)",
    "input_cell_ids": [
      "cell-illinois-feb-2019-290",
      "cell-illinois-feb-2019-300"
    ],
    "input_span_ids": [
      "span-illinois-feb-2019-anchors"
    ],
    "inputs": {
      "p290": 83.0,
      "p300": 90.0,
      "score": 298
    },
    "method": "reviewer sensitivity only: linear interpolation",
    "result_percentile": 88.6,
    "uncertainty": "Neither Illinois nor OpenAI disclosed this interpolation; it cannot be attributed as the launch method."
  },
  "resolved_input_cells": [
    {
      "cell_id": "cell-illinois-feb-2019-290",
      "percentile": 83,
      "score": 290
    },
    {
      "cell_id": "cell-illinois-feb-2019-300",
      "percentile": 90,
      "score": 300
    }
  ]
}
  1. span-calculation-derive-illinois-feb-2019-298accepted derivation and input cells: derive-illinois-feb-2019-298em:dossier-span:sha256:b16e9a841685191e39868f05b3b6e4ab6adbd109794b3af87845e3609e2bfa8d · sha256:76a57ca09947680e56c24258540370c4538e5e7d69931cb76882834e659fb1f2em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-illinois-feb-2019-anchorstotal-scale table, rows 300 and 290em:dossier-span:sha256:1e2736835309f364d7f8553b6e26a7c9802dce86c720c84fa2097f7e3a755ca3 · sha256:29f7ab38f7ff54e4576aa5b3c6fd310debd0520513f68fc3e21744af073923f8em:dossier-edition:sha256:2c2badc4c75c58ca49007bda1f0c1fcad4b301ecd060bc905bd8a741acd80925 · sha256:91185bd6263d73e5976b5b00f28fb9b62301be0bb0347b1d0a308c295f6a6151em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed · no open license confirmed
Calculation · reviewer sensitivity only: linear interpolation: 67.8; equation="p298 = p290 + ((298 - 290) / 10) * (p300 - p290)"; comparison_population=null; depends_on=[].
Relation
relation-assertion-derive-illinois-jul-2018-298 em:dossier-evidence-relation:sha256:76480f3d580abb1b7f822049d98d9d71df986bb9468385f053bc736857135b09
Proposition
em:dossier-proposition:sha256:16d42772ecd2e1cf09e10207fd84bcdd82413184a1ca7e0a6f71222759c24539
Assertion
em:dossier-assertion:sha256:173e036ea146192840c20d9a90986e589e15e92a37c4018264f3dba905e58e4c
{
  "derivation": {
    "derivation_id": "derive-illinois-jul-2018-298",
    "equation": "p298 = p290 + ((298 - 290) / 10) * (p300 - p290)",
    "input_cell_ids": [
      "cell-illinois-jul-2018-290",
      "cell-illinois-jul-2018-300"
    ],
    "input_span_ids": [
      "span-illinois-jul-2018-anchors"
    ],
    "inputs": {
      "p290": 59.0,
      "p300": 70.0,
      "score": 298
    },
    "method": "reviewer sensitivity only: linear interpolation",
    "result_percentile": 67.8,
    "uncertainty": "Neither Illinois nor OpenAI disclosed this interpolation; it cannot be attributed as the launch method."
  },
  "resolved_input_cells": [
    {
      "cell_id": "cell-illinois-jul-2018-290",
      "percentile": 59,
      "score": 290
    },
    {
      "cell_id": "cell-illinois-jul-2018-300",
      "percentile": 70,
      "score": 300
    }
  ]
}
  1. span-calculation-derive-illinois-jul-2018-298accepted derivation and input cells: derive-illinois-jul-2018-298em:dossier-span:sha256:88fbcd4b7b93f462ad98350235a550f2b34f2dbc1a811849f0f40dc7cb03ee18 · sha256:1e02e47e6e3f6a7c2422be3a698ecce3d17899aef055d0f47f2745ec043366f5em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-illinois-jul-2018-anchorstotal-scale table, rows 300 and 290em:dossier-span:sha256:3ec4b97842a7b34ea0a2857a50da7f6a02eb1556351c7ec727b2cccd2743637c · sha256:fe399dcc350821ac8ab546593931f1e34b24d811cad9062126e487a53783a783em:dossier-edition:sha256:2b720aa3b7368249b079f4bc7d3de383efc590510bc33e1ed88db133e9293237 · sha256:a9aa31983924ffec8d0e713ad29fc16a0e466557a57eba5061577afc4698bae6em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed · no open license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-jul-2018-300", "percentile": 70, "score": 300 }, { "cell_id": "cell-illinois-jul-2018-290", "percentile": 59, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-jul-2018-anchors", "supports": "The July chart places the same score region far below the February chart." }
Edition
em:dossier-edition:sha256:2b720aa3b7368249b079f4bc7d3de383efc590510bc33e1ed88db133e9293237
Edition digest
sha256:a9aa31983924ffec8d0e713ad29fc16a0e466557a57eba5061577afc4698bae6
Span
em:dossier-span:sha256:3ec4b97842a7b34ea0a2857a50da7f6a02eb1556351c7ec727b2cccd2743637c
Span digest
sha256:fe399dcc350821ac8ab546593931f1e34b24d811cad9062126e487a53783a783
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-feb-2018-300", "percentile": 90, "score": 300 }, { "cell_id": "cell-illinois-feb-2018-290", "percentile": 85, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-feb-2018-anchors", "supports": "The chart brackets 298 but contains no printed 298 row or interpolation rule." }
Edition
em:dossier-edition:sha256:9be0e724fc35f74c96e635c6263f8f44ef804c3406c34b268bb710b24228a368
Edition digest
sha256:44de7bd89cd4aeacee7b38d7077ff659e3baf552cb9b295401ec9b7229c04964
Span
em:dossier-span:sha256:56a011a09e4e7dd05afbc0309c94d4dcfa1ecf1d6fd3df6a6a6f65204c7ccf03
Span digest
sha256:b3b8f789db7571a4fd8ed73e5380f2847aae920172722b9f6f51eec96c62e9ce
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-feb-2019-300", "percentile": 90, "score": 300 }, { "cell_id": "cell-illinois-feb-2019-290", "percentile": 83, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-feb-2019-anchors", "supports": "This later February chart also brackets 298 but supplies no interpolation rule." }
Edition
em:dossier-edition:sha256:2c2badc4c75c58ca49007bda1f0c1fcad4b301ecd060bc905bd8a741acd80925
Edition digest
sha256:91185bd6263d73e5976b5b00f28fb9b62301be0bb0347b1d0a308c295f6a6151
Span
em:dossier-span:sha256:1e2736835309f364d7f8553b6e26a7c9802dce86c720c84fa2097f7e3a755ca3
Span digest
sha256:29f7ab38f7ff54e4576aa5b3c6fd310debd0520513f68fc3e21744af073923f8
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed
03 The Martínez version of record and code support roughly 45th under the encoded passers model, while its abstract and discussion say roughly 48th; the packet preserves 45/48 as an internal discrepancy.

Typed relation: support

Relation
relation-assertion-claim-martinez-passers-conflict em:dossier-evidence-relation:sha256:98f71b562a1c5d46cd4a56ad684083756ac580d2ecdc0c1db9cb1b0438094252
Proposition
claim-martinez-passers-conflict em:dossier-proposition:sha256:4c6ac4b24cbd908f89d988c4bb1dbc008e9ff3b5bb4fffaa313145d1762465ce
Assertion
assertion-claim-martinez-passers-conflict em:dossier-assertion:sha256:35a975b45cb9ebe6fed26d97fde109f534f7ce69613026271c16226ebf3c1182
Lineage
lineage-martinez-analysis-root em:dossier-lineage:sha256:e6057f0dbfa37d0264e471e67a3941297a56467e67cad715d0eb37490d320cdd

Typed dependence

  1. edge-benchmark-illinois-chartslineage-illinois-comparison-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:3bbbc190e0662f0d0cf22c4bf02a5abe6f9e71e95061afd3dc94f535c83fb57aaccepted_edge_id=edge-benchmark-illinois-charts; accepted_dimension=benchmark; from=lineage-illinois-comparison-root; to=lineage-martinez-analysis-root; effects=It is comparison data, not an additional GPT-4 performance root. | Population drift affects rank without changing the model score. | It is a sensitivity input, not a replication of the model run.
  2. edge-citation-katz-to-martinezlineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:a8c16efd90634c0a152f0fdd7a0a02f8d844004ea05edec24ec4651bf00764ccaccepted_edge_id=edge-citation-katz-to-martinez; accepted_dimension=citation; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=Citation links the analysis lineage to the same reported score.
  3. edge-comparison-class-first-time-passers--1lineage-ncbe-comparison-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:c26568911130747cbc372db3bcb7c6a64cecaa0134fb60ab0def98d7ceb621e4accepted_edge_id=edge-comparison-class-first-time-passers; accepted_dimension=comparison-class; from=lineage-ncbe-comparison-root,lineage-new-york-pass-rate-root; to=lineage-martinez-analysis-root; effects=Comparison populations are not interchangeable. | Administration mix can change percentile without new model evidence. | It parameterizes the comparison model rather than testing GPT-4. | The threshold defines a modeled comparator boundary.
  4. edge-comparison-class-first-time-passers--2lineage-new-york-pass-rate-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:0b7e8f83a90479b952cdbf35c12046e8810bf312a6f413158f9d4567bc45849baccepted_edge_id=edge-comparison-class-first-time-passers; accepted_dimension=comparison-class; from=lineage-ncbe-comparison-root,lineage-new-york-pass-rate-root; to=lineage-martinez-analysis-root; effects=Comparison populations are not interchangeable. | Administration mix can change percentile without new model evidence. | It parameterizes the comparison model rather than testing GPT-4. | The threshold defines a modeled comparator boundary.
  5. edge-data-reported-score-reuselineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:691d48bc190063d50b6e811c7455ec325b1f29bcc7452301694470e83211188aaccepted_edge_id=edge-data-reported-score-reuse; accepted_dimension=data; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=The re-analysis is not a new model-performance experiment. | Score validation changes warrant, not the participant or model-performance root count.
  6. edge-derivation-comparison-inputs--1lineage-ncbe-comparison-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:8e1e29f851db71dfa10a5e236ce568d68a73d9c9dcc9e57be569669606ec66a7accepted_edge_id=edge-derivation-comparison-inputs; accepted_dimension=derivation; from=lineage-ncbe-comparison-root,lineage-new-york-pass-rate-root; to=lineage-martinez-analysis-root; effects=The value is a modeled input, not a GPT-4 observation. | Executable transformations do not create new source or performance roots.
  7. edge-derivation-comparison-inputs--2lineage-new-york-pass-rate-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:4cb023d071772f2c5de160eda970a23d215bd0e51de0ebef48513742b7b92ab4accepted_edge_id=edge-derivation-comparison-inputs; accepted_dimension=derivation; from=lineage-ncbe-comparison-root,lineage-new-york-pass-rate-root; to=lineage-martinez-analysis-root; effects=The value is a modeled input, not a GPT-4 observation. | Executable transformations do not create new source or performance roots.
Calculation · normal CDF at essay score 140 using assumed MBE distribution: 41.520892709392534; equation=null; comparison_population="modeled first-time essay scores"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-first-time-essay em:dossier-evidence-relation:sha256:13748298644aa578769bbabfff2cef7491504ee79bf1d180ece2b059135ed552
Proposition
em:dossier-proposition:sha256:8b4297b5104a65c7f8bddf8b32ed22ff861e9c48bfd3f310cd2004f0422891a3
Assertion
em:dossier-assertion:sha256:cd35ba2db639224fb309527b6345a677d99b6477c0af28d3b1780b3b760db33e
{
  "derivation": {
    "comparison_population": "modeled first-time essay scores",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-first-time-essay",
    "method": "normal CDF at essay score 140 using assumed MBE distribution",
    "result_percentile": 41.520892709392534
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-first-time-essayaccepted derivation and input cells: derive-martinez-first-time-essayem:dossier-span:sha256:7fe07e6a8c3bfc7145de80e008268fc0c0e4c52008cdcc2c56229e50818ae891 · sha256:a9c27ee78ab735acd0cc44a48abf93543d47b7e31106e8b749ad9acd73d94272em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · normal CDF at MBE score 158: 78.82324690739395; equation=null; comparison_population="modeled first-time MBE takers"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-first-time-mbe em:dossier-evidence-relation:sha256:e7b1110a59e6ed3c7a511c4e1e77aa7664a34e076f572963e8d3790a981334bc
Proposition
em:dossier-proposition:sha256:added6c64552253aad1f336b1c4d88de6dd6172647d50fa5bddd3820feecf458
Assertion
em:dossier-assertion:sha256:7df08eee7d48ee98980dc48dcd4257c04f6a03ee5db70b69f16762e4122880ff
{
  "derivation": {
    "comparison_population": "modeled first-time MBE takers",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-first-time-mbe",
    "method": "normal CDF at MBE score 158",
    "result_percentile": 78.82324690739395
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-first-time-mbeaccepted derivation and input cells: derive-martinez-first-time-mbeem:dossier-span:sha256:46a2e94c6b7fa7049cb7e26fb0def120ac9445a8bd95c1b21478c0db1e94424a · sha256:0222f9c217d52a28eaeb28caf33d560864fd14ba99c48d7ebd7fa0aa4fdba995em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · normal CDF at score 298: 61.60252541656707; equation="100 * Phi((298 - 287.6) / derived_ube_sd)"; comparison_population="modeled first-time UBE takers"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-first-time-ube em:dossier-evidence-relation:sha256:a2c43a47e3f6b92d24687c9aa8fc109e6c1395031b0d99748789cacc006a0718
Proposition
em:dossier-proposition:sha256:e239947e9f182874d45854038dd31c8374fadebae4638ef0e25f91f286fe7202
Assertion
em:dossier-assertion:sha256:a5fe847c3d60f4185c2335a2160eb3e8223fb2e1a8e2174b611efd1cc9f442d0
{
  "derivation": {
    "comparison_population": "modeled first-time UBE takers",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-first-time-ube",
    "equation": "100 * Phi((298 - 287.6) / derived_ube_sd)",
    "method": "normal CDF at score 298",
    "result_percentile": 61.60252541656707
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-first-time-ubeaccepted derivation and input cells: derive-martinez-first-time-ubeem:dossier-span:sha256:282eac343e251f80212871bfab6bbad216ef94e8989d5f56387cc19a79d72aca · sha256:95f5718ef7d21ae6fb0218ba919638f6bee7e0a849abfab1144857c83941d8c2em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · analytic reproduction of the executable OSF inputs: {"derived_ube_sd":35.24729455256271,"sample_mbe_sd":17.74327194332649,"z_at_0_27":-0.6128129910166272}; equation=null; comparison_population=null; depends_on=[].
Relation
relation-assertion-derive-martinez-parameters em:dossier-evidence-relation:sha256:592ad3c88ad3d542736bb92ba9aeab1c3f375fd40d9688295eb6e8021a7d88e7
Proposition
em:dossier-proposition:sha256:8a265341b3cca87de499aa7fe9807fcb85405a680c9899fb2844a96717923ac6
Assertion
em:dossier-assertion:sha256:46b0fa65ec0a5d666de7e82ebfe7ea040c7bccf9d1862d67389fdc9ae6f6d6cb
{
  "derivation": {
    "derivation_id": "derive-martinez-parameters",
    "input_cell_ids": [
      "cell-martinez-july-mbe-85",
      "cell-martinez-july-mbe-90",
      "cell-martinez-july-mbe-95",
      "cell-martinez-july-mbe-100",
      "cell-martinez-july-mbe-105",
      "cell-martinez-july-mbe-110",
      "cell-martinez-july-mbe-115",
      "cell-martinez-july-mbe-120",
      "cell-martinez-july-mbe-125",
      "cell-martinez-july-mbe-130",
      "cell-martinez-july-mbe-135",
      "cell-martinez-july-mbe-140",
      "cell-martinez-july-mbe-145",
      "cell-martinez-july-mbe-150",
      "cell-martinez-july-mbe-155",
      "cell-martinez-july-mbe-160",
      "cell-martinez-july-mbe-165",
      "cell-martinez-july-mbe-170",
      "cell-martinez-july-mbe-175",
      "cell-martinez-july-mbe-180",
      "cell-martinez-july-mbe-185",
      "cell-ncbe-ube-score-266",
      "cell-ny-first-timers-rate"
    ],
    "input_span_ids": [
      "span-reshetar-first-time-mean",
      "span-martinez-mean-assumption",
      "span-martinez-script-july-mbe-distribution",
      "span-martinez-script-ube-sd",
      "span-ncbe-ny-cutoff-2022",
      "span-ny-first-timers-2022"
    ],
    "inputs": {
      "assumed_first_time_essay_mean": 143.8,
      "assumed_first_time_ube_mean": 287.6,
      "first_time_mbe_mean": 143.8,
      "july_mbe_binned_observations": 1002,
      "new_york_cutoff": 266.0,
      "new_york_nonpass_proportion": 0.27
    },
    "method": "analytic reproduction of the executable OSF inputs",
    "results": {
      "derived_ube_sd": 35.24729455256271,
      "sample_mbe_sd": 17.74327194332649,
      "z_at_0_27": -0.6128129910166272
    },
    "uncertainty": "The UBE distribution is inferred from aggregate inputs and normality; the essay mean/SD are assumed rather than observed."
  },
  "resolved_input_cells": [
    {
      "cell_id": "cell-martinez-july-mbe-85",
      "count": 2,
      "score": 85
    },
    {
      "cell_id": "cell-martinez-july-mbe-90",
      "count": 2,
      "score": 90
    },
    {
      "cell_id": "cell-martinez-july-mbe-95",
      "count": 5,
      "score": 95
    },
    {
      "cell_id": "cell-martinez-july-mbe-100",
      "count": 6,
      "score": 100
    },
    {
      "cell_id": "cell-martinez-july-mbe-105",
      "count": 13,
      "score": 105
    },
    {
      "cell_id": "cell-martinez-july-mbe-110",
      "count": 22,
      "score": 110
    },
    {
      "cell_id": "cell-martinez-july-mbe-115",
      "count": 33,
      "score": 115
    },
    {
      "cell_id": "cell-martinez-july-mbe-120",
      "count": 56,
      "score": 120
    },
    {
      "cell_id": "cell-martinez-july-mbe-125",
      "count": 73,
      "score": 125
    },
    {
      "cell_id": "cell-martinez-july-mbe-130",
      "count": 78,
      "score": 130
    },
    {
      "cell_id": "cell-martinez-july-mbe-135",
      "count": 104,
      "score": 135
    },
    {
      "cell_id": "cell-martinez-july-mbe-140",
      "count": 96,
      "score": 140
    },
    {
      "cell_id": "cell-martinez-july-mbe-145",
      "count": 101,
      "score": 145
    },
    {
      "cell_id": "cell-martinez-july-mbe-150",
      "count": 99,
      "score": 150
    },
    {
      "cell_id": "cell-martinez-july-mbe-155",
      "count": 99,
      "score": 155
    },
    {
      "cell_id": "cell-martinez-july-mbe-160",
      "count": 79,
      "score": 160
    },
    {
      "cell_id": "cell-martinez-july-mbe-165",
      "count": 64,
      "score": 165
    },
    {
      "cell_id": "cell-martinez-july-mbe-170",
      "count": 38,
      "score": 170
    },
    {
      "cell_id": "cell-martinez-july-mbe-175",
      "count": 22,
      "score": 175
    },
    {
      "cell_id": "cell-martinez-july-mbe-180",
      "count": 8,
      "score": 180
    },
    {
      "cell_id": "cell-martinez-july-mbe-185",
      "count": 2,
      "score": 185
    },
    {
      "cell_id": "cell-ncbe-ube-score-266",
      "column": "Score",
      "row": "266",
      "text": "266"
    },
    {
      "cell_id": "cell-ny-first-timers-rate",
      "column": "Rate",
      "row": "ALL Candidates",
      "text": "73%"
    }
  ]
}
  1. span-calculation-derive-martinez-parametersaccepted derivation and input cells: derive-martinez-parametersem:dossier-span:sha256:b36533d6f13983fa968a987d5263ea1ab6220a87e2b416ab59a0c3d40ea03d09 · sha256:b92c0e5f9e2866c3806a45f478310c538513bb17644f78c91900edf5fc547d2eem:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · normal CDF conditional on modeled essay score >= 135: 15.252536216882072; equation=null; comparison_population="modeled essay scores at or above 135"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-passers-essay em:dossier-evidence-relation:sha256:ab1d937208a7dd4488c51004af267a03414306012ee327ecf10c05d0645a32b7
Proposition
em:dossier-proposition:sha256:6c97aaa7a9bc70f03c07f91df685245b1a1278b97bb4e9bcf56f7e85861dda76
Assertion
em:dossier-assertion:sha256:bfe2fd3438fb2f351efa67ef5066876074b428fd1fcd8b6c379070873fda676b
{
  "derivation": {
    "comparison_population": "modeled essay scores at or above 135",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-passers-essay",
    "method": "normal CDF conditional on modeled essay score >= 135",
    "result_percentile": 15.252536216882072
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-passers-essayaccepted derivation and input cells: derive-martinez-passers-essayem:dossier-span:sha256:75c94b1ed032d669e53b051fa98d8234be91d8062a23ee43147ac17553d9e5db · sha256:e28cf7118b36578f51acca1366d26808d9189b9b699ff2f8ce7c4b77bde9b468em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · normal CDF conditional on modeled MBE score >= 135: 69.31081545343784; equation=null; comparison_population="modeled MBE scores at or above 135"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-passers-mbe em:dossier-evidence-relation:sha256:1dd0e17278567855ddd02c50c7a61cc75f28fc83737928dcc68e610cce4ba6b2
Proposition
em:dossier-proposition:sha256:c564ce54eb3b2d6f72781fee0e99f5162aec79993a6aa87f999d7b8d2775420e
Assertion
em:dossier-assertion:sha256:837a7e443196825e489e5e7f7a09e74cdcd36ce65a43cf63c504a479c59da0a5
{
  "derivation": {
    "comparison_population": "modeled MBE scores at or above 135",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-passers-mbe",
    "method": "normal CDF conditional on modeled MBE score >= 135",
    "result_percentile": 69.31081545343784
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-passers-mbeaccepted derivation and input cells: derive-martinez-passers-mbeem:dossier-span:sha256:25dbcaa16173c2875dcab51c6fbd9210df50cf2aa03cb3e2514724edcc667200 · sha256:bc34a7eda1879f438d7e7945440d3db419371f000787488a1830b7c283f81845em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed
Calculation · normal CDF conditional on modeled UBE score >= 270: 44.45020553281165; equation="100 * (F(298) - F(270)) / (1 - F(270))"; comparison_population="modeled first-time scores at or above 270"; depends_on=["derive-martinez-parameters"].
Relation
relation-assertion-derive-martinez-passers-ube em:dossier-evidence-relation:sha256:20f088f009065cfc3380add119f1dd245d0cc60c26ce7304d2603d810dcceac5
Proposition
em:dossier-proposition:sha256:9119f9ff62ea1e7760aabb53df19e6824e081aaab2d28ddac873b49ba1829c69
Assertion
em:dossier-assertion:sha256:0e84c467e4cdeee807106e9fa021760a1a2e63bcd7fb41c4cfccea488558b23f
{
  "derivation": {
    "comparison_population": "modeled first-time scores at or above 270",
    "depends_on": [
      "derive-martinez-parameters"
    ],
    "derivation_id": "derive-martinez-passers-ube",
    "equation": "100 * (F(298) - F(270)) / (1 - F(270))",
    "method": "normal CDF conditional on modeled UBE score >= 270",
    "result_percentile": 44.45020553281165,
    "uncertainty": "The script uses 270 for this filter after using New York's 266 cutoff to infer UBE SD."
  },
  "resolved_input_cells": []
}
  1. span-calculation-derive-martinez-passers-ubeaccepted derivation and input cells: derive-martinez-passers-ubeem:dossier-span:sha256:3738518581952127b4be8dbb224e03a6eb3126a5ec5c685a837cba9e0d7bfcde · sha256:5cd325ee19d0d60ca21ed01dcf1cc3c6cfb1edfb2f9fb702d680f03a2ec1a8f3em:dossier-edition:sha256:32d3146687af86ae7865d08fbc1636a0c0aa6195e52cb00350ce4b0d525ade2c · sha256:ab330f3d40dfdb5e467da1f66c8f2d318b6b5ed88b770555956d3c5c79151a4bem:dossier-source-work:sha256:bb86be7b07b85a8d09a4fab4b2526a49c82b959ee3fb31bcf33ff7a5e6e27831 · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions · Repository instrumentation under Apache-2.0; accepted source licenses remain attached to their source editions
  2. span-martinez-mean-assumption§3.1, journal p. 588em:dossier-span:sha256:ecc711638dfc4b30115783c7b8d7f32b292b8c723a36f8dfbf421e41923b9461 · sha256:d292fbd6a0a2c16559d69c651b273374cef4c7383b7c2605b97f85381bf8c62dem:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 · sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. span-martinez-script-july-mbe-distributionlines 55-58em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38 · sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0cem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  4. span-martinez-script-ube-sdlines 104-112em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83 · sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55bem:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745 · sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed
  5. span-ncbe-ny-cutoff-2022Minimum Passing UBE Score by Jurisdiction in 2022, row 266em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8 · sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653bem:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081 · sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed · no open license confirmed
  6. span-ny-first-timers-2022FIRST TIMERS, ALL Candidates, Combined 2022em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 · sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f · sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74fem:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed · no open license confirmed
  7. span-reshetar-first-time-meanparagraph beginning 'The numbers from 2019'em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520 · sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49 · sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2em:dossier-source-work:sha256:802fad21dc2e69f5baff90d345e036b898bf958351af8eae39f346d23e005a12 · no open license confirmed · no open license confirmed

percentile_analysis_new.Rmd

discussion, journal p. 598

when examining only those who passed the exam, GPT-4’s performance is estimated to drop to ∼48th percentile overall, and ∼15th percentile on essays.
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:0b68dabbe1d021424ed271a53217094910a82be36032ddf2fb7d527e519ac280
Span digest
sha256:d999f5238c2ce339081283354841c9949e5a6a71a5370ea28eb635484832a707
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

percentile_analysis_new.Rmd

§3.2.2, journal p. 591

With regard to the aggregate UBE score, GPT-4 scored in the ∼45th percentile.
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:e4d9982c357259d2e11c9d6e647569ca0338134240924a766792fbba54de7502
Span digest
sha256:94dbb34cb0b30aefe6ad46a54298be28df4bf1014c3a0d389f63e4615b3dc9bf
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

percentile_analysis_new.Rmd

lines 166-168

{ "format": "code-segment-transcription", "locator": "lines 166-168", "normalization": "exact-code-text", "segments": [ { "segment_id": "segment-martinez-code-filter-ube", "text": "filtered_data_ube <- data_ube[data_ube >= 270]" }, { "segment_id": "segment-martinez-code-filter-mbe", "text": "filtered_data_mbe <- data_mbe[data_mbe >= 135]" }, { "segment_id": "segment-martinez-code-filter-essay", "text": "filtered_data_essay <- data_essay[data_essay >= 135]" } ], "span_id": "span-martinez-script-thresholds", "supports": "The passers analysis filters UBE at 270 even though the SD input uses New York's 266 cutoff." }
Edition
em:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745
Edition digest
sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666
Span
em:dossier-span:sha256:7da66b96a0ba3914388da9f878232040b78dc3be4727b77afe738797c9e53c8b
Span digest
sha256:e4f31613c4c910062cc0c84109333b81d866660c9565c354ac91f5fc1f76716f
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

percentile_analysis_new.Rmd

Table 3, journal p. 591

{ "cells": [ { "cell_id": "cell-martinez-july-ube", "column": "UBE", "row": "July test-takers", "text": "1st\u201368th" }, { "cell_id": "cell-martinez-first-timers-ube", "column": "UBE", "row": "All first-timers", "text": "2nd\u201362rd" }, { "cell_id": "cell-martinez-qualified-attorneys-ube", "column": "UBE", "row": "Qualified attorneys", "text": "0th\u201345th" } ], "format": "table-cell-transcription", "locator": "Table 3, journal p. 591", "normalization": "exact-cell-text", "span_id": "span-martinez-table-45", "supports": "The table reports 45th, not 48th, for the passers/qualified-attorneys comparison." }
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:474fb98347f7e6e5fa04ed79707987e9ee63b657d6f1c5e2435843ec6a1cddbe
Span digest
sha256:200178847adc47281f699728e39c7ce7bb3f84704ed70de68b2e0c012f955be5
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-feb-2018-300", "percentile": 90, "score": 300 }, { "cell_id": "cell-illinois-feb-2018-290", "percentile": 85, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-feb-2018-anchors", "supports": "The chart brackets 298 but contains no printed 298 row or interpolation rule." }
Edition
em:dossier-edition:sha256:9be0e724fc35f74c96e635c6263f8f44ef804c3406c34b268bb710b24228a368
Edition digest
sha256:44de7bd89cd4aeacee7b38d7077ff659e3baf552cb9b295401ec9b7229c04964
Span
em:dossier-span:sha256:56a011a09e4e7dd05afbc0309c94d4dcfa1ecf1d6fd3df6a6a6f65204c7ccf03
Span digest
sha256:b3b8f789db7571a4fd8ed73e5380f2847aae920172722b9f6f51eec96c62e9ce
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-feb-2019-300", "percentile": 90, "score": 300 }, { "cell_id": "cell-illinois-feb-2019-290", "percentile": 83, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-feb-2019-anchors", "supports": "This later February chart also brackets 298 but supplies no interpolation rule." }
Edition
em:dossier-edition:sha256:2c2badc4c75c58ca49007bda1f0c1fcad4b301ecd060bc905bd8a741acd80925
Edition digest
sha256:91185bd6263d73e5976b5b00f28fb9b62301be0bb0347b1d0a308c295f6a6151
Span
em:dossier-span:sha256:1e2736835309f364d7f8553b6e26a7c9802dce86c720c84fa2097f7e3a755ca3
Span digest
sha256:29f7ab38f7ff54e4576aa5b3c6fd310debd0520513f68fc3e21744af073923f8
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Illinois February 2018 Bar Examination Percentile Equivalents

total-scale table, rows 300 and 290

{ "cells": [ { "cell_id": "cell-illinois-jul-2018-300", "percentile": 70, "score": 300 }, { "cell_id": "cell-illinois-jul-2018-290", "percentile": 59, "score": 290 } ], "format": "table-cell-transcription", "locator": "total-scale table, rows 300 and 290", "normalization": "exact-cell-text", "span_id": "span-illinois-jul-2018-anchors", "supports": "The July chart places the same score region far below the February chart." }
Edition
em:dossier-edition:sha256:2b720aa3b7368249b079f4bc7d3de383efc590510bc33e1ed88db133e9293237
Edition digest
sha256:a9aa31983924ffec8d0e713ad29fc16a0e466557a57eba5061577afc4698bae6
Span
em:dossier-span:sha256:3ec4b97842a7b34ea0a2857a50da7f6a02eb1556351c7ec727b2cccd2743637c
Span digest
sha256:fe399dcc350821ac8ab546593931f1e34b24d811cad9062126e487a53783a783
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Appendix for GPT-4 passes the Bar Exam

footnote 5

While we are not fully convinced of the methodological approach taken in some subsequent analysis [78], we do agree that it would be better to consider the raw 297 UBE as falling within a range between 68th and 90th percentile (depending on the precise state and timing of the exam administration).
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:477c595f89b6efc4f4fac04d627e65d95daa7929d10f76d3486a96c6c8bf8b06
Span digest
sha256:7e376175b36f8e2809bc74e378c7354858289a689370b70f82df2965d32b9026
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

First-Time Exam Takers and Repeaters in 2022

table note

NOTE: First-time exam and repeat test taker data supplied by the jurisdictions in this chart are based on those examinees’ testing experience in the reporting jurisdiction only and do not account for possible previous attempts at the bar examination in other jurisdictions.
Edition
em:dossier-edition:sha256:b3dddcd76a7bbe208e296d4b9a13fc53e91cd14df2ae42ee1c91803569240eca
Edition digest
sha256:f42357f4f20e9db92a1a48755cb2f269123cb4ee93cecc042bff83d990989794
Span
em:dossier-span:sha256:f8fe10f696eb9dc37fa067a0b10401f5bf1e29589d6c6071cfaa44c15e59b183
Span digest
sha256:1feddb6e088dae626d7d1cc8f6a0207c19ac7e9f736830bf3a3ae4085e374a42
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

First-Time Exam Takers and Repeaters in 2022

Minimum Passing UBE Score by Jurisdiction in 2022, row 266

{ "cells": [ { "cell_id": "cell-ncbe-ube-score-266", "column": "Score", "row": "266", "text": "266" }, { "cell_id": "cell-ncbe-ube-jurisdictions-266", "column": "Jurisdictions", "row": "266", "text": "Connecticut; District of Columbia; Illinois; Iowa; Kansas; Kentucky; Maryland; Montana; New Jersey; New York; South Carolina; Virgin Islands" } ], "format": "table-cell-transcription", "locator": "Minimum Passing UBE Score by Jurisdiction in 2022, row 266", "normalization": "exact-cell-text", "span_id": "span-ncbe-ny-cutoff-2022", "supports": "New York's 2022 minimum passing UBE score was 266." }
Edition
em:dossier-edition:sha256:2000492e8e5040cd8f5eaa216e7fc1d4d21d4f58afbfc6af5a218f4f53333081
Edition digest
sha256:b55af410679b66b8a0057d5f8dd17baec7985675c94d7af44818f5c4c418cfb7
Span
em:dossier-span:sha256:160241d336f3cc06be5b4d52f6ac9229563916a9ca197dbbacb968b7c60981f8
Span digest
sha256:8e029f660745d0cb0de2acd77e8d6e8c8f9c544d1ba560c3250bc30861ad653b
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

First-Time Exam Takers and Repeaters in 2022

NCBE MBE-based data, February and July 2022 totals

{ "format": "exact-segments", "locator": "NCBE MBE-based data, February and July 2022 totals", "normalization": "collapse-whitespace", "segments": [ { "segment_id": "segment-ncbe-february-repeaters-count", "text": "February likely repeaters taking: 11,289" }, { "segment_id": "segment-ncbe-february-repeaters-percent", "text": "68% of February 2022 examinees were likely repeaters" }, { "segment_id": "segment-ncbe-february-first-timers-count", "text": "February likely first-timers taking: 5,215" }, { "segment_id": "segment-ncbe-february-first-timers-percent", "text": "32% of February 2022 examinees were likely first-time takers" }, { "segment_id": "segment-ncbe-july-repeaters-count", "text": "July likely repeaters taking: 10,200" }, { "segment_id": "segment-ncbe-july-repeaters-percent", "text": "23% of July 2022 examinees were likely repeaters" }, { "segment_id": "segment-ncbe-july-first-timers-count", "text": "July likely first-timers taking: 34,505" }, { "segment_id": "segment-ncbe-july-first-timers-percent", "text": "77% of July 2021 examinees were likely first-time takers" } ], "span_id": "span-ncbe-snapshot-composition", "supports": "February and July have materially different inferred first-time/repeater composition." }
Edition
em:dossier-edition:sha256:5e5c17ba424f84711ba55a14e9bf2b2846c2c05389280d7e21e1f1b2ad37810d
Edition digest
sha256:d2ce119a524286fa88bbaf4438c9052d77a5b728990db41cbc8cad70dc6740c4
Span
em:dossier-span:sha256:352571f2e61d157a8943ca6216ef23f38b6ae5cde1b55838d1a424bcc4ee9f09
Span digest
sha256:c389829939a67dd8c1d353a75cb1bdbc6823160e0582ba17bf83ba6d1e939dbc
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

New York Bar Exam 2022 Statistics

FIRST TIMERS, ALL Candidates, Combined 2022

{ "cells": [ { "cell_id": "cell-ny-first-timers-took", "column": "Took", "row": "ALL Candidates", "text": "9,457" }, { "cell_id": "cell-ny-first-timers-passed", "column": "Passed", "row": "ALL Candidates", "text": "6,867" }, { "cell_id": "cell-ny-first-timers-rate", "column": "Rate", "row": "ALL Candidates", "text": "73%" } ], "format": "table-cell-transcription", "locator": "FIRST TIMERS, ALL Candidates, Combined 2022", "normalization": "exact-cell-text", "span_id": "span-ny-first-timers-2022", "supports": "The combined first-time pass rate was 73%, so the complementary non-pass proportion used by the re-analysis is 27%." }
Edition
em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f
Edition digest
sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74f
Span
em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703
Span digest
sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed

Appendix for GPT-4 passes the Bar Exam

§4(d), Table 7

{ "cells": [ { "cell_id": "cell-katz-gpt4-mbe", "column": "GPT-4", "row": "MBE", "text": "157 points" }, { "cell_id": "cell-katz-gpt4-mee", "column": "GPT-4", "row": "MEE", "text": "84 points" }, { "cell_id": "cell-katz-gpt4-mpt", "column": "GPT-4", "row": "MPT", "text": "56 points" }, { "cell_id": "cell-katz-gpt4-overall", "column": "GPT-4", "row": "overall score", "text": "297 points" } ], "format": "table-cell-transcription", "locator": "\u00a74(d), Table 7", "normalization": "exact-cell-text", "span_id": "span-katz-vor-components", "supports": "The version-of-record component scores sum to 297." }
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:293687df5cf4b74d839237414173205591aacf4850103e1621aece4f21fb763b
Span digest
sha256:e6c808d4b4dcd243210518497a0c8788d06ee91d8571c474c13514c2c8afe04b
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

percentile_analysis_new.Rmd

introduction, journal p. 584

The paper successfully replicates the MBE score of 158, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the essay score (140).
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:47ce609849ae1ff1224df62ce9b453707dd9b7a93cd4413822c84178eac7f853
Span digest
sha256:11db34f6066f861f338266e98c11852743b727734f62c1a773ae257bf219e609
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

percentile_analysis_new.Rmd

lines 55-58

{ "cells": [ { "cell_id": "cell-martinez-july-mbe-85", "count": 2, "score": 85 }, { "cell_id": "cell-martinez-july-mbe-90", "count": 2, "score": 90 }, { "cell_id": "cell-martinez-july-mbe-95", "count": 5, "score": 95 }, { "cell_id": "cell-martinez-july-mbe-100", "count": 6, "score": 100 }, { "cell_id": "cell-martinez-july-mbe-105", "count": 13, "score": 105 }, { "cell_id": "cell-martinez-july-mbe-110", "count": 22, "score": 110 }, { "cell_id": "cell-martinez-july-mbe-115", "count": 33, "score": 115 }, { "cell_id": "cell-martinez-july-mbe-120", "count": 56, "score": 120 }, { "cell_id": "cell-martinez-july-mbe-125", "count": 73, "score": 125 }, { "cell_id": "cell-martinez-july-mbe-130", "count": 78, "score": 130 }, { "cell_id": "cell-martinez-july-mbe-135", "count": 104, "score": 135 }, { "cell_id": "cell-martinez-july-mbe-140", "count": 96, "score": 140 }, { "cell_id": "cell-martinez-july-mbe-145", "count": 101, "score": 145 }, { "cell_id": "cell-martinez-july-mbe-150", "count": 99, "score": 150 }, { "cell_id": "cell-martinez-july-mbe-155", "count": 99, "score": 155 }, { "cell_id": "cell-martinez-july-mbe-160", "count": 79, "score": 160 }, { "cell_id": "cell-martinez-july-mbe-165", "count": 64, "score": 165 }, { "cell_id": "cell-martinez-july-mbe-170", "count": 38, "score": 170 }, { "cell_id": "cell-martinez-july-mbe-175", "count": 22, "score": 175 }, { "cell_id": "cell-martinez-july-mbe-180", "count": 8, "score": 180 }, { "cell_id": "cell-martinez-july-mbe-185", "count": 2, "score": 185 } ], "code_lines": [ { "line_id": "line-martinez-july-distribution-55", "text": "July_distribution <- c(rep(85, 2), rep(90, 2), rep(95, 5), rep(100, 6), rep(105, 13)," }, { "line_id": "line-martinez-july-distribution-56", "text": " rep(110, 22), rep(115, 33), rep(120, 56), rep(125, 73), rep(130, 78)," }, { "line_id": "line-martinez-july-distribution-57", "text": " rep(135, 104), rep(140, 96), rep(145, 101), rep(150, 99), rep(155, 99)," }, { "line_id": "line-martinez-july-distribution-58", "text": " rep(160, 79), rep(165, 64), rep(170, 38), rep(175, 22), rep(180, 8), rep(185, 2))" } ], "format": "code-table-transcription", "locator": "lines 55-58", "normalization": "exact-code-text", "span_id": "span-martinez-script-july-mbe-distribution", "supports": "The executable analysis expands these 21 score-count cells to estimate the July MBE sample standard deviation." }
Edition
em:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745
Edition digest
sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666
Span
em:dossier-span:sha256:c93f04ab5c92ae7955b75278cff1f0392210fc5462823dbe30cba42b5ffbab38
Span digest
sha256:9912e2d4f1e369b4f9529d27b6d1fef5936814f0c6151a9fe9f675af8c375e0c
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

percentile_analysis_new.Rmd

lines 104-112

{ "format": "code-segment-transcription", "locator": "lines 104-112", "normalization": "exact-code-text", "segments": [ { "segment_id": "segment-martinez-code-mean", "text": "mean <- 287.6" }, { "segment_id": "segment-martinez-code-quantile", "text": "quantile_value <- 266" }, { "segment_id": "segment-martinez-code-percentile", "text": "percentile <- 0.27 # 24%" }, { "segment_id": "segment-martinez-code-z-score", "text": "z_score <- qnorm(percentile)" }, { "segment_id": "segment-martinez-code-ube-sd", "text": "sd_UBE <- (quantile_value - mean) / z_score" } ], "span_id": "span-martinez-script-ube-sd", "supports": "The script derives UBE standard deviation from mean 287.6, cutoff 266, and non-pass proportion 0.27." }
Edition
em:dossier-edition:sha256:b135493e681d789459bf386b90f79308cc68da4559b410ad6e818779a81f2745
Edition digest
sha256:c90a873989a59d231475a13a92e3ad1d0d3276514ad54e4e5df695cd35e48666
Span
em:dossier-span:sha256:7c297fee53ebcae4ddc2b4bd2301d9742b570914c364b3993890c92c3ab42d83
Span digest
sha256:29d389334aa2b12045b1d9febb630b66d80d832615d0764534e4e0a2ac49b55b
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

The Testing Column: Why Are February Bar Exam Pass Rates Lower than July Pass Rates?

paragraph beginning 'The numbers from 2019'

The numbers from 2019, the last prepandemic year, are typical: all likely first-time test takers earned an average MBE score of 143.8. Likely repeaters, however, earned an average MBE score of 132.4.
Edition
em:dossier-edition:sha256:852a2ad23d9f6f760f0f21c4147eb6dd1b3c936ea9704e3dc3f743de193e4e49
Edition digest
sha256:460f8d22d10526ee2a92e1fb4045e1cabacb0dc4a668e142a91a30592ccf0ed2
Span
em:dossier-span:sha256:6f0efeaf8fa67729f3f75793e1b64ce9961bb3463e5acc9d9c4a1d5aeb588520
Span digest
sha256:1452bea6f3158b2b58d90800d02a6cf2348935e6a194e07ab22efc88c5f26302
Retrieval
No external retrieval record; repository audit span
License
no open license confirmed · no open license confirmed
04 None of the captured sources measures performance against practicing lawyers or establishes general legal competence.

Typed relation: support

Relation
relation-assertion-claim-no-lawyer-rank em:dossier-evidence-relation:sha256:557b5a29466beafd389b33f1f9c0ab1cf0c96adda878cdc87363397912088a19
Proposition
claim-no-lawyer-rank em:dossier-proposition:sha256:e98f731fe4f066e80cefdb1712238ca5d5b9c08854f86b5a99bd132df05177a6
Assertion
assertion-claim-no-lawyer-rank em:dossier-assertion:sha256:3c3793e084854e911d7a2aece20cf4a7f9f430d3f71a36e87cee09ed9228a514
Lineage
lineage-model-performance-root em:dossier-lineage:sha256:c8b81e09b0245e5c5302dd89be2d627f7658d7ab0d5de6d56adef3df0f65e145

Typed dependence

  1. edge-author-social-collaborationlineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:b616095462050178a4dedd71e5ad524effaa71d20808eceb206d203f9aae8204accepted_edge_id=edge-author-social-collaboration; accepted_dimension=author-social; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=The vendor report and study are socially linked manifestations, not independent tests.
  2. edge-citation-katz-to-martinezlineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:a8c16efd90634c0a152f0fdd7a0a02f8d844004ea05edec24ec4651bf00764ccaccepted_edge_id=edge-citation-katz-to-martinez; accepted_dimension=citation; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=Citation links the analysis lineage to the same reported score.
  3. edge-data-reported-score-reuselineage-model-performance-root → lineage-martinez-analysis-rootem:dossier-evidence-relation:sha256:691d48bc190063d50b6e811c7455ec325b1f29bcc7452301694470e83211188aaccepted_edge_id=edge-data-reported-score-reuse; accepted_dimension=data; from=lineage-model-performance-root; to=lineage-martinez-analysis-root; effects=The re-analysis is not a new model-performance experiment. | Score validation changes warrant, not the participant or model-performance root count.
  4. edge-material-shared-exam-itemslineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:8f61eda50599e5c6c226c2dcb7818c8627f23979fd5b51669aa62e5d52f6d110accepted_edge_id=edge-material-shared-exam-items; accepted_dimension=material; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=All score manifestations inherit the same exam-item materials.
  5. edge-method-single-free-response-runlineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:1ed5b88123954bc8da25cf9422c2df70e30d3569ddd99213e9cf5590bf8522c3accepted_edge_id=edge-method-single-free-response-run; accepted_dimension=method; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=Repeated documents do not supply repeated-run independence.
  6. edge-model-historical-snapshotslineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:e7c0af1eebfe5fa400bcb3369655f40adf482a99cc91a370f6e6a586f847f133accepted_edge_id=edge-model-historical-snapshots; accepted_dimension=model; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=The two snapshots are components of one reported experiment, not two evidence roots.
  7. edge-score-component-compositelineage-model-performance-root → lineage-model-performance-rootem:dossier-evidence-relation:sha256:21278e42f97e114bc1442703ca96eea56725d7bb221a07e5c31f5adfc512b040accepted_edge_id=edge-score-component-composite; accepted_dimension=score; from=lineage-model-performance-root; to=lineage-model-performance-root; effects=297 and 298 are scoring variants within one experiment.

percentile_analysis_new.Rmd

§3.1, journal pp. 588-589

Assuming that UBE scores (as well as MBE and essay subscores) are normally distributed, percentiles of GPT’s score can be directly computed after computing the parameters of these distributions (i.e. the mean and standard deviation).
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:cceeb7bbb8f7496871eeb81724f8be4bcd784bde0cbf28a236ecd81d5ad7b8b3
Span digest
sha256:3a80dd544d7516c8d33d8b55a4cdf44a2975a279ad377d53b09927cfcc26a733
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

GPT-4 Technical Report

abstract, PDF p. 1

passing a simulated bar exam with a score around the top 10% of test takers
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:c574b1bd1adfdbb6b446b321eb656c49dc7a710ed609089c52069dd4da29d296
Span digest
sha256:61393f3895d80b5bb5e4e4240ff6c4b8bb9246f22b58ac1a766643c63887e95a
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

GPT-4 Technical Report

Appendix A.1, PDF p. 23

The Uniform Bar Exam was run by our collaborators at CaseText and Stanford CodeX.
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:61f2be1da866e5764042a9022c4a0a0eb5d320a0df71f95a24a1a1eb5fd271ac
Span digest
sha256:e0b72dc61e4614a425e991b3fd9641db637b3516311659728f615bed0ea4100c
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

Appendix for GPT-4 passes the Bar Exam

footnote 5

While we are not fully convinced of the methodological approach taken in some subsequent analysis [78], we do agree that it would be better to consider the raw 297 UBE as falling within a range between 68th and 90th percentile (depending on the precise state and timing of the exam administration).
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:477c595f89b6efc4f4fac04d627e65d95daa7929d10f76d3486a96c6c8bf8b06
Span digest
sha256:7e376175b36f8e2809bc74e378c7354858289a689370b70f82df2965d32b9026
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

Appendix for GPT-4 passes the Bar Exam

§4(d), Table 7

{ "cells": [ { "cell_id": "cell-katz-gpt4-mbe", "column": "GPT-4", "row": "MBE", "text": "157 points" }, { "cell_id": "cell-katz-gpt4-mee", "column": "GPT-4", "row": "MEE", "text": "84 points" }, { "cell_id": "cell-katz-gpt4-mpt", "column": "GPT-4", "row": "MPT", "text": "56 points" }, { "cell_id": "cell-katz-gpt4-overall", "column": "GPT-4", "row": "overall score", "text": "297 points" } ], "format": "table-cell-transcription", "locator": "\u00a74(d), Table 7", "normalization": "exact-cell-text", "span_id": "span-katz-vor-components", "supports": "The version-of-record component scores sum to 297." }
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:293687df5cf4b74d839237414173205591aacf4850103e1621aece4f21fb763b
Span digest
sha256:e6c808d4b4dcd243210518497a0c8788d06ee91d8571c474c13514c2c8afe04b
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

percentile_analysis_new.Rmd

introduction, journal p. 584

The paper successfully replicates the MBE score of 158, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the essay score (140).
Edition
em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667
Edition digest
sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
Span
em:dossier-span:sha256:47ce609849ae1ff1224df62ce9b453707dd9b7a93cd4413822c84178eac7f853
Span digest
sha256:11db34f6066f861f338266e98c11852743b727734f62c1a773ae257bf219e609
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; no file license confirmed; no node or file license confirmed · CC BY 4.0; no file license confirmed; no node or file license confirmed

Appendix for GPT-4 passes the Bar Exam

§3(a), materials

{ "format": "exact-segments", "locator": "\u00a73(a), materials", "normalization": "collapse-whitespace", "segments": [ { "segment_id": "segment-katz-mee-mpt-materials", "text": "For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination." }, { "segment_id": "segment-katz-mbe-materials", "text": "The MBE questions used in this study are official multistate bar examination questions from previous administrations of the UBE [65]." } ], "span_id": "span-katz-vor-materials", "supports": "The study components share disclosed exam-item materials and administration lineage." }
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:076f2679614413d88a366ceec578a4c96672df24e2fe9dc8afe1242c8c7a5496
Span digest
sha256:9107c310f1bb38d087940d7432c989d2ab81d8e79356bc6a42a6c64bbf2db37e
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

GPT-4 Technical Report

Appendix A.3, PDF p. 24

we simply ran these free response questions each only a single time at our best-guess temperature (0.6) and prompt
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:8b5e9985c61e0fea14503003ec6422ec54340f4069def5d98d5e1b7147da6ba6
Span digest
sha256:eebbf2c0c2999d72a017e9af6b4799d65577fb0715ac5d85e6ee757e05aae78e
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

GPT-4 Technical Report

Appendix A.6, PDF p. 25

We ran GPT-4 multiple-choice questions using a model snapshot from March 1, 2023, whereas the free-response questions were run and scored using a non-final model snapshot from February 23, 2023.
Edition
em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb
Edition digest
sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
Span
em:dossier-span:sha256:6f6629457faeeb750d379bc919a2030743b0fd07db285521e3a54a46f02ab3ab
Span digest
sha256:9dc5f4260c333122ddb37e7ca7e2f3eecd1d1976475145759b772a7701c58f1e
Retrieval
No external retrieval record; repository audit span
License
arXiv non-exclusive distribution license; no Creative Commons license confirmed · arXiv non-exclusive distribution license; no Creative Commons license confirmed

Appendix for GPT-4 passes the Bar Exam

footnote 4

Best prompt and/or hyperparameter combination on the MBE would push this score to 298 or higher. Here, we report the MBE average of 75.7% which composites to a 297.
Edition
em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b
Edition digest
sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
Span
em:dossier-span:sha256:5dd561ae54601a051ef573af4ec859665f4f1d2fb801788a1339de8b16dd44fc
Span digest
sha256:0ce9516e5b95f3b35121d471b8ab8d39b67199ffb40941deb28e07e2a07926ab
Retrieval
No external retrieval record; repository audit span
License
CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree

Verify every number

Complete count ledgers

Every displayed total is a view over these typed members; no total is maintained as marketing copy.

1 · Historical Model Run
  1. root-historical-simulated-ube-run Historical simulated-UBE model-performance root
    Member
    root-historical-simulated-ube-run
    Object type
    lineages
    Object key
    lineage-model-performance-root
    Object ID
    em:dossier-lineage:sha256:c8b81e09b0245e5c5302dd89be2d627f7658d7ab0d5de6d56adef3df0f65e145
    Status
    known
    Dependence
    data, method, model, social
    Basis span
    em:dossier-span:sha256:ee862ec363b34142a0bebb11247959c3457a8982de69653364d7d36f10059703 sha256:57920d8649f509df54908d08dfaefb9860e1afc005095c6cd4cb051344dff044
    Basis edition
    em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
    Basis work
    em:dossier-source-work:sha256:880443d56eafdb224149d953f7de48e7550d53f814008551895b180ecb4b203b · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree
3 · Official Administration Sensitivities
  1. root-illinois-february-2018 Illinois February 2018 comparison root
    Member
    root-illinois-february-2018
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:56a011a09e4e7dd05afbc0309c94d4dcfa1ecf1d6fd3df6a6a6f65204c7ccf03 sha256:b3b8f789db7571a4fd8ed73e5380f2847aae920172722b9f6f51eec96c62e9ce
    Basis edition
    em:dossier-edition:sha256:9be0e724fc35f74c96e635c6263f8f44ef804c3406c34b268bb710b24228a368 sha256:44de7bd89cd4aeacee7b38d7077ff659e3baf552cb9b295401ec9b7229c04964
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
  2. root-illinois-february-2019 Illinois February 2019 comparison root
    Member
    root-illinois-february-2019
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:1e2736835309f364d7f8553b6e26a7c9802dce86c720c84fa2097f7e3a755ca3 sha256:29f7ab38f7ff54e4576aa5b3c6fd310debd0520513f68fc3e21744af073923f8
    Basis edition
    em:dossier-edition:sha256:2c2badc4c75c58ca49007bda1f0c1fcad4b301ecd060bc905bd8a741acd80925 sha256:91185bd6263d73e5976b5b00f28fb9b62301be0bb0347b1d0a308c295f6a6151
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
  3. root-illinois-july-2018 Illinois July 2018 comparison root
    Member
    root-illinois-july-2018
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:3ec4b97842a7b34ea0a2857a50da7f6a02eb1556351c7ec727b2cccd2743637c sha256:fe399dcc350821ac8ab546593931f1e34b24d811cad9062126e487a53783a783
    Basis edition
    em:dossier-edition:sha256:2b720aa3b7368249b079f4bc7d3de383efc590510bc33e1ed88db133e9293237 sha256:a9aa31983924ffec8d0e713ad29fc16a0e466557a57eba5061577afc4698bae6
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
7 · Rooted Evidence Units
  1. root-model-performance Historical model-performance root
    Member
    root-model-performance
    Object type
    lineages
    Object key
    lineage-model-performance-root
    Object ID
    em:dossier-lineage:sha256:c8b81e09b0245e5c5302dd89be2d627f7658d7ab0d5de6d56adef3df0f65e145
    Status
    known
    Dependence
    data, method, model, social
    Basis span
    em:dossier-span:sha256:ee862ec363b34142a0bebb11247959c3457a8982de69653364d7d36f10059703 sha256:57920d8649f509df54908d08dfaefb9860e1afc005095c6cd4cb051344dff044
    Basis edition
    em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
    Basis work
    em:dossier-source-work:sha256:880443d56eafdb224149d953f7de48e7550d53f814008551895b180ecb4b203b · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree
  2. root-martinez-reanalysis Martinez modeled re-analysis root
    Member
    root-martinez-reanalysis
    Object type
    lineages
    Object key
    lineage-martinez-analysis-root
    Object ID
    em:dossier-lineage:sha256:e6057f0dbfa37d0264e471e67a3941297a56467e67cad715d0eb37490d320cdd
    Status
    known
    Dependence
    data, method, model, source
    Basis span
    em:dossier-span:sha256:e4d9982c357259d2e11c9d6e647569ca0338134240924a766792fbba54de7502 sha256:94dbb34cb0b30aefe6ad46a54298be28df4bf1014c3a0d389f63e4615b3dc9bf
    Basis edition
    em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
    Basis work
    em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. root-illinois-feb-2018 Illinois February 2018 root
    Member
    root-illinois-feb-2018
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:56a011a09e4e7dd05afbc0309c94d4dcfa1ecf1d6fd3df6a6a6f65204c7ccf03 sha256:b3b8f789db7571a4fd8ed73e5380f2847aae920172722b9f6f51eec96c62e9ce
    Basis edition
    em:dossier-edition:sha256:9be0e724fc35f74c96e635c6263f8f44ef804c3406c34b268bb710b24228a368 sha256:44de7bd89cd4aeacee7b38d7077ff659e3baf552cb9b295401ec9b7229c04964
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
  4. root-illinois-feb-2019 Illinois February 2019 root
    Member
    root-illinois-feb-2019
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:1e2736835309f364d7f8553b6e26a7c9802dce86c720c84fa2097f7e3a755ca3 sha256:29f7ab38f7ff54e4576aa5b3c6fd310debd0520513f68fc3e21744af073923f8
    Basis edition
    em:dossier-edition:sha256:2c2badc4c75c58ca49007bda1f0c1fcad4b301ecd060bc905bd8a741acd80925 sha256:91185bd6263d73e5976b5b00f28fb9b62301be0bb0347b1d0a308c295f6a6151
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
  5. root-illinois-jul-2018 Illinois July 2018 root
    Member
    root-illinois-jul-2018
    Object type
    lineages
    Object key
    lineage-illinois-comparison-root
    Object ID
    em:dossier-lineage:sha256:a5b9820aa7d6c9062b89944f259392c039fbe5285b626d4f71d54dfd44f51a20
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:3ec4b97842a7b34ea0a2857a50da7f6a02eb1556351c7ec727b2cccd2743637c sha256:fe399dcc350821ac8ab546593931f1e34b24d811cad9062126e487a53783a783
    Basis edition
    em:dossier-edition:sha256:2b720aa3b7368249b079f4bc7d3de383efc590510bc33e1ed88db133e9293237 sha256:a9aa31983924ffec8d0e713ad29fc16a0e466557a57eba5061577afc4698bae6
    Basis work
    em:dossier-source-work:sha256:a2e64d0c97109c831bb2facc8c180da8179ed37526502ea73dbe3016fdf003dd · no open license confirmed
  6. root-ncbe-aggregates NCBE scoring and population aggregate root
    Member
    root-ncbe-aggregates
    Object type
    lineages
    Object key
    lineage-ncbe-comparison-root
    Object ID
    em:dossier-lineage:sha256:a674ad6aa3453c54eb437d844fd6070006c25bbb05cb6463bfffd2eee9e24eae
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:b00f1cab5f375334f432b9de6ffb480cebae7ee1557ad424feb6a6223ae1cf25 sha256:0f6d015b519414ebec9cd4f12bbf7dea62dc9f77fb21378d73690b451d5f0f1e
    Basis edition
    em:dossier-edition:sha256:3b7b0ad7942bb25d248c4fffe8e6bcfa1b806e7f695c9cda6ffd170359cc1ba6 sha256:ab04a05f122d9dff51b76f2e9be9f78c0bdcf25f58fc62047cc4c5abd040346c
    Basis work
    em:dossier-source-work:sha256:324982bb2b90295b84f9e309ab84ca9a46cd0f1be9e9e3079a18bcc20bf58e63 · no open license confirmed
  7. root-new-york-pass-rates New York first-time pass-rate root
    Member
    root-new-york-pass-rates
    Object type
    lineages
    Object key
    lineage-new-york-pass-rate-root
    Object ID
    em:dossier-lineage:sha256:b426c4edd6b881b995272c355d5dc869ffe4153c0bcbe662e7ffef1a5267e48b
    Status
    known
    Dependence
    data, source
    Basis span
    em:dossier-span:sha256:468763dc341321dd638ef9a08f6606ac5e02806a4086e4b0ba7bc05400c1d703 sha256:e55680bc3d09d5c2968a8449c122db232a89c5f89b29dc9944b69c2b1eeeee11
    Basis edition
    em:dossier-edition:sha256:10bbf9ff899ba3ad12de154e0ab9ea7cc3f34c0f2263e79951e26539911bec5f sha256:bbdcb28fb38b0d7d03ddb4e4bcfbcca170d2443491a180fee0978c7510cdb74f
    Basis work
    em:dossier-source-work:sha256:51aa3262776f5da4a8ef12ab7f49366ef761951213e3a38f519a0a6b5d6534f1 · no open license confirmed
3 · Unresolved Boundaries
  1. boundary-launch-comparator Launch comparison distribution is unspecified
    Member
    boundary-launch-comparator
    Object type
    assertions
    Object key
    assertion-claim-launch-comparison-unspecified
    Object ID
    em:dossier-assertion:sha256:e5b09f98417934e590aa916b7e5c657dc707fb55ab33c81d87a6a724d48fc159
    Basis span
    em:dossier-span:sha256:ca6ba91cbc68c2a3aa343fc059aa3058d27baa364e998d9c05393578c6045d5c sha256:53c5c3cd65f0c06ae12ec339f9a8f503b977ebaf9c1cd8723d9caecfa42bcdcb
    Basis edition
    em:dossier-edition:sha256:21621ede6a23bb3f98867c0918e5d9808d6dced5ca3f271a3a52e561810f2f7b sha256:9d10cfedd531e62d96ea27bb78b00a9148e6b5d8135807836c5982e65a42416e
    Basis work
    em:dossier-source-work:sha256:880443d56eafdb224149d953f7de48e7550d53f814008551895b180ecb4b203b · CC BY 4.0; CC BY 4.0 at the article/file level; collection-level license null; no open license confirmed; no repository license at the pinned tree
  2. boundary-45-48 Martinez 45th-versus-48th internal discrepancy
    Member
    boundary-45-48
    Object type
    assertions
    Object key
    assertion-claim-martinez-passers-conflict
    Object ID
    em:dossier-assertion:sha256:35a975b45cb9ebe6fed26d97fde109f534f7ce69613026271c16226ebf3c1182
    Basis span
    em:dossier-span:sha256:e4d9982c357259d2e11c9d6e647569ca0338134240924a766792fbba54de7502 sha256:94dbb34cb0b30aefe6ad46a54298be28df4bf1014c3a0d389f63e4615b3dc9bf
    Basis edition
    em:dossier-edition:sha256:65219a4731cb2e79f58629db4793e659a48da74aa74c3a3db5049ce260701667 sha256:c74c66e3adc8a45af5bac9e338c11d927dd2d95c8386b2d4bb52ef2591c7a0a5
    Basis work
    em:dossier-source-work:sha256:3e91b5cabaf08b648f9e9e505d7fa5e9519cca7becf3ec753b1777075e4d512b · CC BY 4.0; no file license confirmed; no node or file license confirmed
  3. boundary-lawyer-quality No practicing-lawyer comparison
    Member
    boundary-lawyer-quality
    Object type
    assertions
    Object key
    assertion-claim-no-lawyer-rank
    Object ID
    em:dossier-assertion:sha256:3c3793e084854e911d7a2aece20cf4a7f9f430d3f71a36e87cee09ed9228a514
    Basis span
    em:dossier-span:sha256:c574b1bd1adfdbb6b446b321eb656c49dc7a710ed609089c52069dd4da29d296 sha256:61393f3895d80b5bb5e4e4240ff6c4b8bb9246f22b58ac1a766643c63887e95a
    Basis edition
    em:dossier-edition:sha256:859fa96f390685d21c4db9633a6d9c7fec7abe76cdf8a1f8c0c9c82501b193eb sha256:1703b719a1fad06284c9cde691496e2ee6cc9371ab7808d8abfc50ab3ed0c2f9
    Basis work
    em:dossier-source-work:sha256:2c6c231a62e11f961467d990d3bd0ad2d4d4568fcd39891c16e6fe4622e9448b · arXiv non-exclusive distribution license; no Creative Commons license confirmed

Five-line lexicon

The units are different on purpose

Comparison population
The people or scores against which a percentile is calculated.
Administration sensitivity
How the same score ranks under a different exam date and population.
Historical model run
One past evaluation setup, not a claim about a current product.
Modeled re-analysis
A calculation using disclosed assumptions and reused source values.
Boundary
A missing comparator, conflict, or scope limit that remains visible.

Build receipt

Reproduce this projection

Reproducible projection
Dossier
em:dossier:sha256:babe89ba3bda594a8d9f2db86a5a2987f284437a069b940d19b6928856d936d1
View policy
epistemedia-skeptical-v1
Catalog
em:catalog:sha256:4ad9ec81da535bc3654c54fda56a78fbb9f225a2241b59046bf04f275737db35
Frontier
em:frontier:sha256:f02db233ff58fb7bda690ac78335d18a3b36046b578f44ca54d4bdc291821f47
Accepted commit
a5c025f37672be0c6e8b315aa9dd719d8a4f68d1
Epistemic policy
commons-balanced-v0.1
Disclosure policy
public-noninterference-v0.1
Compiler
epistemedia/0.2.0
Content digest
ac6f1ee94ad6f315b0a688e631aecdc52c6615cea0e9d1e759dc508326d120a1