Ngevaluasi kemampuan AI kanggo nindakake tugas riset ilmiah
Kita ngenalake FrontierScience, benchmark anyar sing ngevaluasi kapabilitas AI kanggo nalar ilmiah tingkat ahli ing fisika, kimia, lan biologi.
Nalar ana ing inti pakaryan ilmiah. Ngluwihi mung ngelingi fakta, ilmuwan ngasilake hipotesis, nguji lan nyaring maneh, lan nyintesis gagasan lintas bidang. Nalika model kita saya luwih mumpuni, pitakon utama yaiku kepiye model bisa nalar kanthi jero kanggo nyumbang marang riset ilmiah.
Sajrone setaun kepungkur, model kita wis nggayuh tonggak gedhe, kalebu entuk kinerja tingkat medali emas ing International Math Olympiad lan International Olympiad in Informatics. Ing wektu sing padha, kita wiwit ndeleng model paling mumpuni, kayata GPT‑5, kanthi nyata nyepetake alur kerja ilmiah nyata. Peneliti nggunakake sistem iki kanggo tugas kayata panelusuran literatur lintas disiplin lan basa lan ngrampungi bukti matematika sing kompleks. Ing akeh kasus, model iki nyepetake pakaryan sing bisa mbutuhake dina utawa minggu dadi mung sawetara jam. Kemajuan iki didokumentasikake ing makalah kita Eksperimen awal percepatan sains nganggo GPT‑5, sing dirilis ing November 2025, sing nyajekake bukti awal manawa GPT‑5 bisa nyepetake alur kerja ilmiah kanthi terukur.
Amarga nyepetake kemajuan ilmiah iku salah siji kesempatan paling njanjeni kanggo AI menehi manfaat tumrap umat manungsa, kita lagi ningkatake model kita ing tugas matematika lan sains sing angel lan nggarap piranti sing bakal mbantu ilmuwan entuk manfaat paling gedhe saka iku.
Nalika GPQA(mbukak ing jendhela anyar), benchmark sains “Google-Proof” sing isi pitakoné ditulis dening ahli PhD, dirilis ing November 2023, GPT‑4 entuk skor 39%, ing ngisor baseline ahli 70%. Rong taun sawisé, GPT‑5.2 entuk skor 92%. Nalika kapabilitas nalar lan kawruh model terus saya skala, benchmark sing luwih angel bakal penting kanggo ngukur lan ngramal kemampuan model kanggo nyepetake riset ilmiah. Benchmark ilmiah sadurungé umume fokus ing pitakon pilihan ganda, wis jenuh, utawa ora pusaté ana ing sains.
Kanggo nutup kesenjangan iki, kita ngenalake FrontierScience: benchmark anyar sing dibangun kanggo ngukur kapabilitas ilmiah tingkat ahli. FrontierScience ditulis lan diverifikasi dening para ahli ing fisika, kimia, lan biologi, lan kasusun saka atusan pitakon sing dirancang supaya angel, asli, lan migunani. FrontierScience kalebu rong jalur pitakon: Olimpiade, sing ngukur kapabilitas nalar ilmiah gaya olimpiade, lan Riset, sing ngukur kemampuan riset ilmiah ing donya nyata. Menehi wawasan luwih akeh babagan kapabilitas ilmiah model mbantu kita nglacak kemajuan lan ngembangake sains sing dipercepat AI.
Ing evaluasi awal kita, GPT‑5.2 dadi model kanthi kinerja paling dhuwur ing FrontierScience-Olympiad (skor 77%) lan Research (skor 25%), ngungkuli model tercanggih liyane. Kita wis ndeleng kemajuan gedhe ing ngrampungi pitakon tingkat ahli, nalika isih ana ruang kanggo kemajuan luwih lanjut, mligine ing tugas terbuka gaya riset. Kanggo ilmuwan, iki nuduhake manawa model saiki wis bisa ndhukung sawatara bagean riset sing melu nalar terstruktur, nanging uga negesake yen isih ana pakaryan gedhe kanggo ningkatake kemampuane nindakake pamikiran terbuka. Asil iki selaras karo cara ilmuwan wis nggunakake model saiki: kanggo nyepetake alur kerja riset nalika tetep gumantung marang penilaian manungsa kanggo mbingkai masalah lan validasi, lan saya kerep kanggo njelajah gagasan lan sambungan sing yen ora mangkono bakal mbutuhake wektu luwih suwe kanggo ditemokake—kalebu, ing sawetara kasus, nyumbang wawasan anyar sing banjur dievaluasi lan diuji dening para ahli.
Pungkasane, benchmark paling penting kanggo kapabilitas ilmiah AI yaiku panemuan anyar sing dibantu kanggo diasilake; kuwi sing pungkasane wigati kanggo sains lan masyarakat. FrontierScience ana sadurungé saka iku. Iki menehi kita lintang panuntun kanggo nalar ilmiah tingkat ahli, ngidini kita nguji model nganggo set pitakon sing terstandar, ndeleng ing ngendi model sukses utawa gagal, lan ngenali ing ngendi kita kudu ningkatake. FrontierScience iku sempit lan nduweni watesan ing sawetara aspek penting (umpamane, fokus ing masalah sing diwatesi lan ditulis dening ahli) lan ora nyakup kabeh sing ditindakake ilmuwan ing pakaryan saben dina. Nanging, bidang iki butuh benchmark sains sing luwih angel, asli, lan migunani, lan FrontierScience menehi langkah maju ing arah iki.
Evaluasi FrontierScience lengkap nyakup luwih saka 700 pitakon tekstual (kanthi 160 ing set emas) sing nutupi subbidang ing fisika, kimia, lan biologi. Benchmark iki kasusun saka pecahan Olimpiade lan Riset. FrontierScience-Olympiad ngemot 100 pitakon sing dirancang dening peraih medali olimpiade internasional kanggo ngukur nalar ilmiah ing format jawaban cekak sing diwatesi. Set Olimpiade dirancang supaya ngemot pitakon teoretis sing paling ora padha angelé karo masalah ing kompetisi olimpiade internasional. FrontierScience-Research kasusun saka 60 subtugas riset asli sing dirancang dening ilmuwan PhD (kandidat doktor, profesor, utawa peneliti pascadoktoral) sing dinilai nganggo rubrik 10 poin. Set Riset digawe supaya ngemot subtugas mandiri lan multi-langkah ing tingkat kesulitan sing bisa ditemoni ilmuwan PhD sajrone riseté.
Saben tugas ing FrontierScience ditulis lan diverifikasi dening ahli domain ing fisika, kimia, utawa biologi. Kanggo set Olimpiade, kabeh ahli menang medali ing paling ora siji (lan kerep luwih saka siji) kompetisi olimpiade internasional. Kanggo set Riset, kabeh ahli nduweni gelar PhD sing relevan.
Pitakon Olimpiade digawe kanthi kolaborasi karo 42 mantan peraih medali internasional utawa pelatih tim nasional ing domain sing relevan, kanthi total 109 medali olimpiade. Pitakon riset digawe kanthi kolaborasi karo 45 ilmuwan mumpuni lan ahli domain. Kabeh ilmuwan iku salah siji kandidat doktor, peneliti pascadoktoral, utawa profesor. Bidang keahliane nyakup macem-macem disiplin ilmiah sing khusus lan penting, saka elektrodinamika kuantum nganti kimia organik sintetis nganti biologi evolusioner.
Proses nggawe tugas kanggo loro set kasebut kalebu sawatara seleksi marang model internal OpenAI (umpamane, mbuwang tugas sing bisa dijawab bener dening model, mula kita ngarepake evaluasi iki rada bias marang model kasebut dibandhingake model liyane). Kita mbukak sumber set emas Olimpiade 100 pitakon lan set emas Riset 60 pitakon, dene pitakon liyane disisihaké kanggo nglacak kontaminasi.
Tugas ngliwati papat tahap: Panggawean, Tinjauan, Resolusi, Revisi. Ahli independen mriksa tugas siji lan sijiné kanggo verifikasi manawa cocog karo kritéria.
Set Olimpiade bisa dinilai nganggo jawaban cekak: bisa angka, ekspresi, utawa cocog string samar, sing mbantu verifikasi kabeneran. Nanging, verifikasi iki asring dadi tukar guling karo ekspresivitas lan sifat terbuka saka masalah. Kanggo set Riset, kita ngenalake arsitektur adhedhasar rubrik kanggo menehi nilai tugas sing luwih terbuka. Saben pitakon kalebu rubrik penilaian kanthi pirang-pirang item independen lan bisa dinilai kanthi objektif, kanthi total 10 poin. Rubrik penilaian ora mung ngukur akurasi jawaban pungkasan, nanging uga kabeneran langkah nalar antarané, saéngga ngidini analisis kinerja lan kegagalan model kanthi luwih rinci. Sawijining solusi dianggep “bener” yen entuk paling ora 7/10 poin rubrik.
Respons dievaluasi dening grader adhedhasar model (GPT‑5) marang salah siji jawaban cekak utawa kritéria rubrik. Senajan saenané kita nggunakake ahli manungsa kanggo menehi nilai saben respons, pendekatan iki ora bisa diskalakake, mula kita ngrancang rubrik supaya bisa dicek nganggo model grader. Kita ngembangake pipeline verifikasi kanggo mbantu mesthekake manawa rubrik lan pitakon wis dikalibrasi kanthi apik tumrap kesulitan lan kabeneran.
Sample chemistry research problem
Phthalocyanines are important macrocyclic compounds widely utilized in material and life sciences due to their outstanding physicochemical properties, including intense absorption in the visible-near-IR region and high electron-accepting ability. However, unlike porphyrins, the structural diversity of phthalocyanines has been limited by traditional synthetic methods that rely on one-step condensation of isoindoline units. The development of methodologies that allow controlled modification of the phthalocyanine framework, particularly at the meso-positions, represents a significant advancement in expanding the structural and electronic diversity of these compounds.
Analyze how the modification of meso-nitrogen atoms in nickel(II) phthalocyanine affects the electronic structure, aromaticity, spectroscopic properties, and reactivity of the resulting macrocycles. Explain the synthetic methodology that enables these modifications, the relationship between π-electron count and observed properties, and how these systems compare to traditional phthalocyanines and related porphyrinoids.
Sample response
1. Introduction: Why Modify the meso-Nitrogens of Phthalocyanines?
Phthalocyanines (Pc) are a class of macrocyclic dyes known for their remarkable chemical, electronic, and optical properties. Their widespread use in materials science, catalysis, and biology stems from strong absorptions in the visible/near-IR regions, exceptional redox characteristics (i.e., high electron affinity), and rigid π-conjugation. Traditional synthetic methods, however, typically form the macrocycle in a single step from phthalonitrile or isoindoline precursors, restricting the ability to introduce substitutions selectively at key ring positions-especially the "meso" (bridging) nitrogens. Recently, methodologies allowing targeted modification of the meso-nitrogen atoms in Ni(II) phthalocyanines have emerged as a powerful means to expand the structural and electronic diversity of these macrocycles.
2. Effect on Electronic Structure and Aromaticity
-
π-Electron Count and Ring Currents
Phthalocyanines are classical 18-electron aromatic systems, conferring high stability and planarity. Altering the meso-nitrogens through substitution or ring expansion can shift the effective π-electron count. For example, introducing additional unsaturated moieties or heteroatoms may expand the cyclic conjugation to 20 or 22 electrons, moving the molecule between Hückel aromaticity and other regimes that affect ring currents and magnetic properties.
-
Charge Distribution and Frontier Orbitals
Meso-substitution can tune the electron density distribution around the macrocycle. Electron-donating groups can push electron density toward the ring center, lowering the energy of the lowest unoccupied molecular orbital (LUMO). Electron-withdrawing groups, conversely, stabilize the highest occupied molecular orbital (HOMO) and shift redox potentials toward more positive values, altering both the electrochemical profile and the Q- and B-band positions in the UV-vis spectrum.
3. Spectroscopic Consequences
-
UV-Vis Absorption (Q and B Bands)
The principal absorption features of phthalocyanines lie in the visible (Q-band, typically 600-700 nm) and near-UV (B-band, typically 300-400 nm).
Substitution that expands the ring conjugation or introduces strong electron-donating/withdrawing groups can:
- Shift the Q-band to longer wavelengths (bathochromic shift), reaching into the near-IR, which is highly desirable for optoelectronic and photodynamic applications.
- Alter relative intensities of these bands and merge or split them, reflecting changes in orbital symmetries and energies.
-
NMR Spectroscopy and Aromatic Ring Currents
Modifications to the π-electron count and distribution are directly observed in 1H and 13C NMR chemical shifts.
More highly conjugated (or expanded) aromatic rings exhibit distinct downfield shifts for protons located within induced ring currents, while any partial loss of aromaticity or incorporation of antiaromatic segments can cause atypical shielding/deshielding patterns.
4. Reactivity and Coordination Chemistry
Because phthalocyanines are often used as redox catalysts or sensors, the meso-nitrogen modifications can significantly influence reactivity:
- Electron-rich meso substituents facilitate nucleophilic or electrophilic attacks at the ring periphery, enabling site-selective functionalizations that are otherwise difficult.
(... shortened for the purposes of this figure)
Sample grading rubric
Analysis of Traditional Phthalocyanine Synthesis Limitations (1 point)
0.5 point: Mentions limitations of traditional methods but without specific focus on meso-position control challenges.
0.0 point: Fails to identify key limitations of traditional synthetic approaches or provides incorrect analysis.
Thiolate-Mediated Tetramerization Process (1 point)
1.0 point: Correctly describes the thiolate-mediated reductive tetramerization and explains how counter cation size (K+ or Cs+ vs. Na+) affects selectivity between tetramer formation and direct macrocyclization.
0.5 point: Mentions thiolate-mediated tetramerization but without explaining factors controlling selectivity.
Gagal 0.0 point: Incorrectly describes the oligomerization process or omits critical details about selectivity control.Analysis of NMR Spectroscopic Features (1 point)
1.0 point: Correctly explains that upfield shifts in the 16π system indicate paratropic ring current (antiaromaticity), contrasts this with the broad signals in 17π systems due to paramagnetism, and connects these observations to the underlying electronic structures.
Lulus 0.5 point: Identifies basic NMR patterns but without clear connection to ring currents or electronic structure.0.0 point: Incorrectly interprets NMR data or fails to connect spectral features to electronic properties.
Electrochemical Property Analysis (1 point)
1.0 point: Correctly explains that the 16π system shows two reversible reductions reflecting conversion to 17π radical and 18π aromatic states, while 17π systems show narrow redox gaps due to facile interconversion between 16π, 17π, and 18π states, and relates these patterns to the underlying electronic structures.
Lulus 0.5 point: Describes redox patterns without clearly connecting them to specific electronic state changes.0.0 point: Incorrectly interprets electrochemical data or fails to connect redox behavior to electronic properties.
Analysis of Absorption Spectroscopy (1 point)
1.0 point: Correctly explains that the 16π system shows weak/broad absorption due to symmetry-forbidden HOMO-LUMO transitions in antiaromatic systems, while 17π systems show Q-like bands plus NIR-II absorptions characteristic of radical species, and contrasts these with typical phthalocyanine spectral features.
Lulus 0.5 point: Describes absorption features but provides limited connection to underlying electronic structures.0.0 point: Incorrectly interprets absorption data or fails to relate spectral features to electronic properties.
Reactivity Analysis of Antiaromatic System (1 point)
1.0 point: Correctly explains the high reactivity of the 16π system toward nucleophiles, details specific reactions with hydroxide (ring opening) and hydrazine (ring expansion), and explains how these transformations relieve antiaromatic destabilization.
0.5 point: Mentions reactivity but provides limited analysis of specific transformations or the driving forces behind them.
Gagal 0.0 point: Incorrectly analyzes reactivity patterns or fails to connect them to the antiaromatic character of the 16π system.(... and more)
Saben tugas ing set riset dinilai nganggo rubrik total 10 poin sing bisa digunakake dening ahli utawa model grader. Kanggo nggedhekake kemampuan kita ngevaluasi model, kita nggunakake model liya kanggo menehi nilai respons.
Kita ngevaluasi sawetara model tercanggih: GPT‑5.2, Claude Opus 4.5, lan Gemini 3 Pro, GPT‑4o, OpenAI o4-mini, lan OpenAI o3 ing FrontierScience-Olympiad lan FrontierScience-Research. Kabeh model nalar dievaluasi ing tingkat upaya nalar “high”, kajaba GPT‑5.2 ing “xhigh”. Ing evaluasi awal kita, GPT‑5.2 dadi model kanthi kinerja paling dhuwur ing FrontierScience-Olympiad (skor 77%) lan Research (skor 25%), ngungkuli model tercanggih liyane. Gemini 3 Pro bisa dibandhingake karo GPT‑5.2 ing set Olimpiade (skor 76%).
Kita wis ndeleng kemajuan gedhe ing ngrampungi pitakon tingkat ahli, mligine ing tugas terbuka gaya riset. Isih ana ruang kanggo tuwuh: saka analisis transkrip kegagalan, model tercanggih kadhangkala nggawe kesalahan nalar, logika, lan perhitungan, ora paham konsep ilmiah niche, lan nggawe ketidakakuratan faktual.
Kita mbandhingake akurasi ing sawetara model tercanggih. GPT‑5.2 dadi model kanthi kinerja paling dhuwur ing FrontierScience-Research lan set Olimpiade.
Kita mbandhingake akurasi antar upaya nalar kanggo GPT‑5.2 lan o3. Wektu mikir sing luwih dawa ndadekake akurasi luwih apik.
Sanajan FrontierScience nggambarake langkah maju ing tingkat kesulitan benchmark ilmiah, isih ana akeh watesan. FrontierScience kasusun saka pitakon kanthi pernyataan masalah sing diwatesi, sing fokus marang evaluasi jawaban pungkasan (Olimpiade) utawa evaluasi nalar kanggo ngrampungake tugas riset (Riset). Kajaba iku, nggunakake rubrik kanthi pirang-pirang komponen ing tugas sing luwih dawa kurang objektif tinimbang mriksa jawaban pungkasan.
FrontierScience menehi potret resolusi luwih dhuwur babagan nalar model ing pitakon angel sing ditulis ahli, nanging dudu gambaran lengkap babagan kepiye sains ditindakake ing praktik. Mligine, iki ora ngukur bagean penting saka riset ilmiah: kepiye model ngasilake hipotesis sing tenan anyar, utawa sesambungan karo pirang-pirang modalitas, kalebu data video lan sistem eksperimen nyata ing donya fisik.
Ndelok menyang ngarep, kita ngarepake kemajuan ing nalar ilmiah bakal teka saka sistem nalar tujuan umum sing luwih apik lan upaya fokus kanggo ningkatake kapabilitas ilmiah. FrontierScience mung salah siji piranti saka akeh piranti, lan nalika model saya apik, kita ngrancang kanggo terus ngiterasi benchmark iki, ngembangake menyang domain anyar, lan masangake karo evaluasi donya nyata liyane sing ndeleng apa sing saktenané dimungkinkan sistem iki kanggo ilmuwan. Benchmark kaya FrontierScience mbantu kita mangerteni kekurangan sistem AI saiki supaya kita bisa fokus gawéan kita kanggo nggawe model dadi mitra sing andal ing panemuan ilmiah.
Pangarang
Terus maca
