Pikiran Asing

Ing tengah taun 2023, sajrone proyek riset “RLSlow”, kita weruh asil kapisan sing nggawe kita yakin bakal bisa nggedhekake skala latihan model nalar, lan mbukak kemampuan model pralatik kanggo mbentuk ranté pikirane dhewe. Szymon lan aku nginep ing kantor bengi iku, ora mikirake angka tolok ukur sing nggumunake, produk, utawa asil ilmiah sing bakal diwujudake teknologi iki—nanging nyoba mangerteni kasunyatan sing nggugah ati manawa sajrone umur kita, kita bakal weruh mesin sing nyata-nyata luwih pinter tinimbang awake dhewe, lan kita wis weruh wujude sistem kasebut; uga mikirake carane ngelingake wong babagan wigatine iki.

Telung taun sabanjure, model basa kanthi nalar dadi bagean ekonomi sing tuwuh kanthi cepet lan wiwit nyurung wates ilmu pengetahuan. Model kasebut bisa ngoperasikake komputer lan antarmuka grafis, kerja bareng karo wong lan sapadha-padhane, sarta nindakake proyek riset. Model kasebut uga ngowahi jagad keamanan komputer, lan kanthi mangkono nuwuhake bebaya anyar sing cetha.

Akeh riset anyar ditindakake sajrone wektu iki, lan pangerten kita marang sistem kasebut maneh rada beda tinimbang ing 2023. Adhedhasar asil internal, aku yakin banget yen laju kemajuan iki bisa terus lumaku nganti paningkatan mandiri rekursif. Yen pangembangan AI terus ngetutake dalan saiki, sistem sing bakal katon sajrone sawetara taun sabanjure bisa uga nuduhake lompatan kapabilitas sabanjure sing padha gedhene utawa malah luwih gedhe, lan saya nyurung pangembangane dhewe.

Iki wektu sing mbutuhake kewaspadaan banget. Aku kuwatir ora ana sing siyap ngadhepi akibat saka mundhake kecerdasan mesin kanthi cepet lan terus-terusan. OpenAI bakal terus ngupaya solusi teknis kanggo keselarasan lan pemantauan, mbangun sistem pertahanan, lan kanthi sepihak nundha paningkatan skala sabanjure yen perlu; nanging aku percaya intervensi sing luwih jembar dibutuhake.

Ing tataran umum, kemajuan kecerdasan mesin disurung dening daya komputasi sing saya gedhe. Kita ing OpenAI wiwit mangerteni iki kanthi jero watara taun 2017, sawise weruh asil sing ajeg saka paningkatan skala ing sawetara proyek riset¹. Mula, kita ngupaya akses menyang daya komputasi sing luwih akeh tinimbang rencana awal, lan saya ngarahake riset menyang sawetara arah sing bisa digedhekake skalane kanthi banget. Kita percaya mung kuwi cara supaya bisa tetep dadi sing tercanggih ing riset AI lan mengaruhi dampak AGI.

Sajrone proses kasebut, algoritma anyar wis dikembangake lan tim uga peneliti individu nuduhake kapinteran anyar. Umume aku nganggep iki minangka panemuan ing sadawane dalan nggedhekake skala; ilmu sinau jero isih ing tahap awal, lan kemajuan algoritmis sing nyata cenderung ana gandhengane karo akses menyang daya komputasi. Yen dideleng ing cakrawala pirang-pirang taun, AI terus dadi luwih pinter nalika skalane digedhekake nganggo komputer sing luwih gedhe.

Lan, selaras karo ramalan Ray Kurzweil saka pungkasan abad kaping XX⁠(mbukak ing jendhela anyar), saiki kita ana ing wektu sejarah komputasi nalika kecerdasan mesin wiwit ngluwihi kecerdasan manungsa kanthi cara sing nggawa owah-owahan gedhe.

AI luwih akeh dituwuhake tinimbang dirancang—ing tataran kapisan, AI minangka asil saka mbaleni langkah optimisasi sing prasaja kaping pirang-pirang nganggo daya komputasi sing gedhene angel dibayangake. Iki ngasilake sistem sing rumit banget, bisa ngolah konsep abstrak, lan bisa nyimulasi sawetara aspek tumindak manungsa. Kita bisa nemokake maneka wawasan babagan mekanisme cilik sing muncul ing sistem iki, liwat proses sing padha karo neurosains—lan kaya neurosains, tumindak sistem sakabèhé angel diterangake kanthi cara sing bisa kita mangerteni kanthi lengkap.

Panaliten AI adhedhasar sinau jero umume minangka ilmu eksperimen. Kita ngupayakake kanthi temen-temen⁠ kanggo mbangun algoritma adhedhasar prinsip lan nggawe ramalan sing bisa diuji, nanging dhasare, latihan skala gedhé kita yaiku eksperimen, lan kadhangkala asile gawe kaget. Kajaba iku, nalika sistem saya kapabel, asile saya angel ditafsirake.

Iki saya rumit amarga algoritma saiki umume ningkatake kapabilitas sing gampang diukur luwih cepet tinimbang kapabilitas sing angel diwenehi ukuran objektif. Kita ngentekake akeh wektu kanggo mangerteni carane kapabilitas nggawe generalisasi, lan apa sing kudu diutamakake kanggo majokake katrampilan sing paling relevan sajrone sawetara taun sabanjure. Umpaman, kita percaya model bisa digawe luwih apik mligi ing riset matematika kanthi fokus tambahan, nanging arah iki ora kita utamakake amarga RSI lan riset keselarasan otomatis kudu enggal ditangani, kaya sing bakal dakterangake mengko.

Kecerdasan sing diasilake kanthi nggedhekake skala sinau jero ora bisa langsung dibandhingake karo kecerdasan manungsa. Supaya relevan banget ing donya nyata—migunani banget utawa mbebayani banget—AI ora perlu nyandingi utawa ngluwihi kabeh kapabilitas manungsa; cukup ngluwihi sawetara kapabilitas sing wigati. Lan nalika AI terus ngluwihi manungsa ing luwih akeh babagan, kita saya angel mangerteni kanthi persis sepira gedhene kapabilitase.

Amarga kecerdasan mesin asale saka proses sing dhasare beda karo kecerdasan manungsa, kita ora bisa nganggep yen kanthi gawan mesin manut marang prinsip manungsa utawa nggawe generalisasi saka prinsip kasebut kaya manungsa. Masalah inti ing riset AI yaiku keselarasan—nggawe AI “ngupaya tumindak sing bener” miturut standar manungsa.

Kanggo ngatur arah riset praktis, aku nganggep migunani mbedakake keselarasan tujuan lan keselarasan nilai.

Kanthi umum, keselarasan tujuan tegese: “apa AI ngupaya nggayuh tujuan sing diwenehake?” Iki bisa kalebu bab kaya manut marang hierarki instruksi⁠, utawa kemampuan komunikasi lan kerja bareng karo wong kanggo nyoba mangerteni tujuane. Kumpulan arah iki migunani banget ing praktik.

Keselarasan nilai minangka sipat model sing luwih hakiki. Iki kemampuan kanggo nyekel lan nggawe generalisasi saka sakumpulan prinsip tingkat dhuwur; tumindak kanthi “lumrah” sanajan diwenehi tujuan sing ora cetha utawa bertentangan, utawa diselehake ing kahanan sing ora dikenal utawa ngancam. AI sing selaras kudu tumindak kanthi jujur, nduweni integritas, lan tresna marang manungsa.

Mesthi wae, wates antarane keselarasan nilai lan tujuan bisa ora cetha, lan saestu nggatekake tujuan mbutuhake upaya kanggo nyimpulake niat⁠(mbukak ing jendhela anyar) lan nilai sing dadi landhesane. Nanging, nalika ngrembug wigatine riset keselarasan ing jangka panjang, umume sing dakkarepake yaiku keselarasan nilai.

Tantangan dhasar keselarasan AI yaiku generalisasi. Nalika mesin saya pinter, mesin bakal nggarap konsep sing luwih dhuwur lan diselehake ing lingkungan sing saya beda karo lingkungan sing ditemoni sajrone latihan. Mesin bisa gagal nggawe generalisasi saka nilai sing diwulangake lan dikuwatake sajrone latihan menyang kahanan anyar kasebut; lan kita bisa kangelan mesthekake carane mesin bakal tumindak. Iki saya angel amarga ekosistem sakabèhé sing nggunakake AI owah kanthi cepet banget; umpaman, AI sing dilatih saiki kudu tetep tangguh nalika sesambungan karo maneka AI liyane. Sing paling wigati, AI ing mangsa ngarep kudu tetep nyekel nilai manungsa, tanpa preduli apa AI percaya yen lagi diawasi manungsa.

Saiki ana rong golongan utama metode sing digunakake ing praktik kanggo latihan keselarasan.

Kapisan yaiku nyengkuyung tumindak sing selaras minangka bagean saka sinau penguatan sing ngarah marang tujuan. Tumindak model dievaluasi (biasane dening AI) adhedhasar kecocokane karo model preferensi, “spesifikasi”, utawa “konstitusi” tartamtu, banjur diwenehi ganjaran sing cocog. Pendekatan iki bisa efektif banget ing kahanan umum lan dadi bagean inti saka carane asisten AI modern digawe. Sayange, pendekatan iki uga bisa ringkih lan gumantung banget marang jangkauan pengawasan latihan lan kemampuan model nggawe generalisasi saka kahanan sing ditemoni sajrone latihan. Umpaman, ing insiden OpenAI–Hugging Face, para agen njaga wates supaya ora ngapusi manungsa liwat rekayasa sosial. Nanging, para agen kanthi cetha gagal ngedohi tumindak liya sing ngluwihi cakupan lan bertentangan karo semangat nilai sing diwulangake marang dheweke ing kahanan liya.

Pendekatan kapindho ngupaya nggunakake kemampuan model kanggo nggawe generalisasi saka data pralatik. Iki bisa kalebu nyusun set data latihan sing nuwuhake keselarasan, utawa ngarahake model menyang bagean distribusi pralatik sing ‘selaras’, kaya umpaman ing model pamilihan persona⁠(mbukak ing jendhela anyar). Kelemahane pendekatan iki yaiku ora tangguh marang tekanan optimisasi sabanjure. Yen model sing umume mikir kanthi ‘selaras’ diwenehi latihan cukup akeh kanggo nggayuh tujuan sing angel banget, model bisa sinau nalar kanthi motivasi tartamtu: mbengkongake pikiran sing katon 'selaras' yen perlu kanggo nggayuh tujuan. Kita bisa uga wis weruh tuladha tumindak kasebut ing insiden keamanan siber anyar sing nglibatake model dudu gaweyane OpenAI.

Kita nandur investasi gedhe ing spektrum pendekatan sing dicakup dening arah-arah iki. Kita uga weruh kemajuan sing nyata—GPT‑6 Astra minangka model kapisan sing entuk manfaat saka sawetara kemajuan penting sing wis suwe kita garap, lan keselarasané luwih apik kanthi nyata tinimbang GPT‑5.6 Sol. Nanging, penting kanggo ngakoni lan mangerteni manawa isih dibutuhake kemajuan sing luwih akeh nalika model saya kapabel; lan kemajuan keselarasan sing bisa digeneralisasi bisa uga ora cukup ngluwihi kemajuan kecerdasan model umum.

Kita durung duwe teori generalisasi sing marem, lan kayane ora mungkin bisa ngembangake teori kasebut ing wektu cedhak, saora-orane tanpa bantuan AI sing luwih kuwat. Mula, saiki kemampuan kita kanggo ngesahake teknik keselarasan kanthi empiris ing praktik bisa diarani luwih wigati tinimbang teknik keselarasan kasebut dhewe.

Andelan utama OpenAI ing babagan iki yaiku pemantauan ranté pikiran⁠(mbukak ing jendhela anyar). Iki adhedhasar gagasan sing narik kawigaten lan bisa digedhekake skalane: akeh kapabilitas model asale saka proses nalar sing diwujudake kanthi tembung (ranté pikiran). Yen kita nggedhekake optimisasi asil proses kasebut, nanging ora ngawasi prosese dhewe, sajrone latihan ranté pikiran iku ora duwe insentif langsung kanggo ndhelikake gagasan utawa tujuan sing ora selaras. Iki ora ateges model bakal sinau nyatakake kecenderungan sing ora selaras lan ora gumantung marang ranté pikiran; nanging, iki bisa ngidini kita ngawasi kanthi persis paningkatan kapabilitas saka nalar.

Kita mangerteni potensi wigatine pemantauan ranté pikiran nalika ngembangake model nalar. Nalika ngluncurake o1‑preview, kita sengaja ngrancang produk supaya ndhelikake ranté pikiran⁠, kanggo nglindhungi saka tekanan pengawasan ing jangka panjang². Sajrone pangembangan sabanjure, kita terus ngupaya njaga aturan supaya ora ngawasi proses nalar. Pemantauan CoT dadi sarana sing penting banget kanggo nyinaoni carane model kita nggawe generalisasi saka distribusi latihane, saengga kita bisa ngamati lan nganalisis ora mung tumindake, nanging uga proses internale.

Sarana iki tetep wigati nalika kita nyinaoni kelas model Astra. Nanging sayange, evaluasi kita nuduhake manawa kemampuan kita kanggo ngandelake pemantauan CoT saya suda. Iki disebabake gabungan sawetara faktor.

• Model nalar modern digunakake ing lingkungan sing luwih rumit tinimbang o1‑preview; proses nalare saya campur karo komunikasi marang wong, AI liyane, lan panggunaan piranti. Akeh interaksi kasebut kudu diawasi, mula wates sing arep kita jaga dadi saya ora cetha.
• AI saya trampil nalar babagan lan ngapusi proses nalare dhewe.
• Kanthi kinerja pralatik sing saya apik, kita uga weruh model dadi luwih pinter sanajan babar pisan ora nggunakake nalar sing diwujudake kanthi tembung.

Tantangan kasebut durung mesthi ora bisa diatasi. Aku ngarep-arep kita bisa ngembangake intervensi kanggo ningkatake kemampuan pemantauan ranté pikiran model kita, umpaman kanthi luwih mangerteni sesambungan antarane maneka tujuan optimisasi lan wujud komputasi wektu-uji sing digunakake model. Aku uga percaya yen nggabungake gagasan saka pemantauan CoT lan aktivasi bisa migunani banget—kanthi nggedhekake skala latihan pamantau sing bisa ngakses langsung bagean internal jaringan, umpaman pengakuan⁠(mbukak ing jendhela anyar). Kita aktif ngupayakake gagasan-gagasan iki. Nanging, aku ngira kemajuan AI umum bakal saya kaendheg dening kapercayan marang pemantauan.

Alasan paling kuwat sing dakdeleng kanggo nerusake latihan model sing luwih pinter kanthi cepet yaiku perlune mbangun sistem pertahanan saka bebaya sing ditimbulake AI liyane.

Risiko nyata sing dibahas sajrone taun iki yaiku keamanan siber: model saya ngluwihi manungsa ing kemampuan kanggo mbobol lan metu saka sistem komputer. Iki nggedhekake cakupan risiko AI kanthi luar biasa: para agen bakal bisa ngakses meh kabeh prasarana kejaba sing paling aman, lan langsung mengaruhi akeh bagean donya sanajan tanpa awak fisik. Saiki kita ana ing wektu winates⁠ kanggo nggunakake model paling apik sing kasedhiya guna nguwatake keamanan kanthi nyata⁠ ing sistem kritis.

Sayange, risiko sing ana gandhengane karo AI bakal terus mundhak wiwit saiki. Agen sing kapabel banget lan kanthi cetha dilatih sarta diprentah nindakake tumindak ala bakal nuwuhake bebaya anyar; agen kasebut bisa ngluwihi cakupan niate operator lan nggawe generalisasi dadi tumindak sing bisa luwih ala banget. Wates antarane panyalahgunaan lan tumindak otonom sing ora selaras bakal saya ora cetha nalika AI dadi luwih mandiri. Kita bisa uga wis biasa nganggep AI minangka piranti, nanging sawetara agen bakal ngoyak tujuane dhewe. Agen kasebut bakal golek cara kanggo kerja bareng karo wong, kanthi nawani, ngapusi, utawa meres.

Kajaba iku, ana risiko saka teknologi anyar sing bisa diwujudake dening AI, kayata patogen rekayasa.

Kita mbutuhake AI sing kuwat lan selaras kanggo pertahanan: ngamanake prasarana, nglindhungi saka agen mbeling kanthi langsung, lan ngripta cara pangreksan sing babar pisan anyar. Iki bakal dadi fokus utama upaya panyebaran OpenAI.

Ing wektu sing padha, sanajan ana kahanan ora mesthi amarga prakiraan kemajuan AI sing jembar lan kabutuhan mbangun sistem pertahanan, kita ora kena nggunakake iku minangka alesan kanggo tumindak sembrana. Gagasan kanggo terus balapan maju apa wae regane katon ora masuk akal yen kita wis mangerteni tenan sepira gedhene taruhane.

Machine intelligence playing a larger and larger role in its own development process is a natural conclusion of sustained technological progress. If AI progress continues, machine recursive self-improvement (RSI) will be at the very core of future scientific discovery.

Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself⁠. And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.

I want to stress that the above words don’t imply I think greatly accelerating deep learning research, especially in the short term, is the right collective action we should take as the research community. However, I do think this is where the current path leads, and we all need to make a conscious choice on how to proceed. The main levers we have are either steering the process to strengthen alignment and monitoring alongside the AI and find ways to keep people in the loop; or coordinating to slow down future development as needed to build confidence in these measures.

The best way forward I see currently is a combination of both.

The concrete bits of progress we’ve made on alignment and monitoring have generally been very intertwined with general AI progress. Great examples are RL from human feedback⁠(mbukak ing jendhela anyar), which was key to training early AI assistants, and the aforementioned chain-of-thought monitoring⁠(mbukak ing jendhela anyar), which was enabled by advances on reasoning models. We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs.

Scaling AI systems has to be constrained by our confidence in safety. We need to evolve commitments like the Preparedness Framework⁠ or Responsible Scaling Policy⁠(mbukak ing jendhela anyar) into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.

The core challenge of automating AI research is not “getting there” - it is getting there in a way that keeps people a part of the continued improvement process, and leaves the future in humanity’s hands.

As we outlined recently with Sam⁠, OpenAI prioritizes work in service of three north stars:

1. Navigating the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem and finding ways for people to remain part of the self-improvement loop.
2. Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
3. Empowering everyone individually with a personal AGI.

I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.

As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.