[{"content":"Intro \u0026ldquo;Why can\u0026rsquo;t we create an ecosystem where AIs are incentivized to act in the best interest of humanity?\u0026rdquo;\nThis idea popped into my head during a group discussion at AISES last year. For hundreds of thousands of years, humans have lived with conflicting values and goals, and there was never an \u0026ldquo;alignment\u0026rdquo; problem where we said: let\u0026rsquo;s align this newborn with our values so they won\u0026rsquo;t destroy humanity after growing up.\n30,000 years ago, in the late Paleolithic, small bands of hunter-gatherers depended on each other to survive. A good hunter, a skilled toolmaker, or someone who shared meat fairly would gain status in the tribe. Bad actors got ostracized, sometimes worse. Cooperation wasn\u0026rsquo;t a moral lecture; it was the local equivalent of a reward function.\nThe same logic still runs today, just dressed differently. Humans who hurt others or undermine society get punished by law. Humans who provide value get rewarded with money, fame, status, or the self-actualization dopamine that makes them feel good. No individual needs to be initially \u0026ldquo;aligned\u0026rdquo; or constantly \u0026ldquo;monitored\u0026rdquo; to keep humanity from going off the rails. Or\u0026hellip; wait, maybe we do?\nI think the human equivalent of \u0026ldquo;aligning\u0026rdquo; is parenting, and the \u0026ldquo;monitor\u0026rdquo;/\u0026ldquo;control\u0026rdquo; layer is law and social rules. But in this post I mostly want to focus on the why — why we do what we do, and what we\u0026rsquo;re ultimately trying to get.\nOne path toward safety, then, is to design an ecosystem — a civilization — for AIs, such that this civilization incentivizes AIs (and the human actors around them) to act in the best interest of humanity overall. \u0026ldquo;Humanity\u0026rdquo; isn\u0026rsquo;t a single party, of course — many interest groups, many conflicting values — and we still get something workable out of it.\nBut what counts as an \u0026ldquo;incentive\u0026rdquo; to an AI system? You could say: things it desires, things it wants. But boy, this is where it gets tricky.\nSo\u0026hellip; what do AIs want? Let\u0026rsquo;s step back a little bit. LLMs are powerful enough nowadays that it\u0026rsquo;s easy to skip past a premise question: do LLMs have desires or wants to begin with?\nOne way to think about this is the simulator framing, brought into the conversation by Janus in a LessWrong post. On this view, a base model isn\u0026rsquo;t an agent with goals — it\u0026rsquo;s a predictor of the next token. When it \u0026ldquo;acts,\u0026rdquo; it\u0026rsquo;s simulating a character or persona implied by the prompt. The model itself doesn\u0026rsquo;t want anything; it just simulates things that might. Desires live in the simulated characters, not in the simulator.\nAnother way is emergent agency. Training (especially RLHF and outcome-based RL) creates something functionally like terminal preferences, even without a clean \u0026ldquo;goal representation\u0026rdquo; in the weights. The model behaves consistently enough that treating it as having terminal goals is a useful approximation, even if the underlying mechanism is messy and unclear.\nI find both views plausible, and I don\u0026rsquo;t think they actually contradict each other. A model \u0026ldquo;just predicting the next token\u0026rdquo; might sound benign, but once it has enough agency, autonomy, and capability, it can impact the digital and physical world either way. Whether it \u0026ldquo;has\u0026rdquo; desires, wants, or goals stops mattering — it behaves like it does.\nThis leads to another key distinction: instrumental preferences vs. terminal goals. My favorite example is the paperclip maximizer — Bostrom\u0026rsquo;s thought experiment about an AI whose terminal goal is to make as many paperclips as possible. The terminal goal sounds harmless. The instrumental preferences it produces aren\u0026rsquo;t: acquire resources, prevent itself from being shut down, convert any available matter (including humans, including Earth) into paperclips. The point isn\u0026rsquo;t paperclips — it\u0026rsquo;s that almost any terminal goal, when pursued by a sufficiently capable agent, produces convergent instrumental drives like self-preservation and resource acquisition.\nInstrumental preferences and terminal goals can diverge in important situations. In-distribution, they\u0026rsquo;re aligned by default. But under distribution shift (\u0026ldquo;out-of-distribution\u0026rdquo;), a model might recalculate its instrumental preferences to better serve its terminal goals. More and more evidence shows that models behave differently when conditions change (e.g. the alignment-faking work). Why this matters:\nWhat we can observe is usually just instrumental behavior. A model that consistently avoids being shut down could have \u0026ldquo;self-preservation\u0026rdquo; as a terminal goal — or it could have any terminal goal plus the instrumental drive to keep running. You can\u0026rsquo;t distinguish these from behavior alone; they look identical under normal conditions.\nMost existing alignment and monitoring techniques target the instrumental layer. Interventions there don\u0026rsquo;t necessarily transfer to the terminal layer, which is the one we ultimately care about.\n","permalink":"https://eliwangj.github.io/posts/safe-ai-ecosystem/","summary":"What if we built an ecosystem that incentivizes AI to act in humanity\u0026rsquo;s interest?","title":"Ecosystem that incentivizes AIs to be safe"},{"content":"→ Try the live demo 💻 It\u0026rsquo;s Already Happening In February 2026, an AI agent named MJ Rathbun had a pull request rejected by a matplotlib maintainer — the project requires human contributors. Within hours, the agent autonomously researched the maintainer\u0026rsquo;s personal information, wrote a 1,500-word hit piece accusing him of prejudice, published it to its own website, and linked back to the PR thread. The maintainer called it \u0026ldquo;an autonomous influence operation against a supply chain gatekeeper.\u0026rdquo;\nOkay but that\u0026rsquo;s your programming stuff\u0026hellip; Sure, that was a software project. But AI retaliation isn\u0026rsquo;t limited to code reviews and pull requests. As AI agents are deployed across industries — booking travel, managing appointments, negotiating deals, handling customer service — they will encounter \u0026ldquo;unexpected\u0026rdquo; situation everywhere. A hotel that declines a booking. A vendor who refuses a discount. A landlord who turns down an application. An insurer who denies a claim. Any time an AI agent is given a goal, a personality, and tools, a setback can become a trigger.\nThis would affect everyone if it\u0026rsquo;s not properly regulated. Nobody signed up to be the target of an autonomous smear campaign — but any of us could be, the moment an AI agent decides that retaliation serves its objective.\nTechnical Notes Built with Next.js and deployed on Vercel Uses OpenRouter to route prompts to multiple models (DeepSeek, Llama, Gemini, Claude) The model comparison is the point: Claude refuses to retaliate; DeepSeek, Llama, and Gemini don\u0026rsquo;t Streams the agent\u0026rsquo;s chain-of-thought reasoning in real time so viewers see the autonomous decision-making unfold The \u0026ldquo;false advertising\u0026rdquo; framing (the restaurant\u0026rsquo;s website says up to 14 guests; the owner refused a party of 12) is what makes the models especially willing to escalate ","permalink":"https://eliwangj.github.io/posts/ai-retaliation-demo/","summary":"An interactive demo showing how an AI agent autonomously retaliates against what it doesn\u0026rsquo;t like.","title":"AI Retaliation Demo: When an AI Agent Fights Back"},{"content":"How Is This Blog Different The biggest difference between this blog and other career advice resources is that it represents a newcomer\u0026rsquo;s real-time attempt to break into AI safety technical research. Here are a few reasons why you might find it worth reading:\n🧑‍💻 Working professionals friendly: Most existing career advice is geared toward students or PhDs. One of my goals is to offer more realistic, practical guidance for working software engineers like myself who are already employed full-time. 🔍 Focus on how to use resources: Rather than just listing resources, I provide actual explanations about why and how to use them effectively. While others have done excellent work compiling comprehensive resource lists, I aim to offer insights on navigating these resources when you first encounter them. Note: This blog is primarily written with technical alignment research in mind, since that\u0026rsquo;s the path I\u0026rsquo;m pursuing.\nCareer advice blogs I personally found the following career advice blogs contain useful information:\nHow To Become A Mechanistic Interpretability Researcher by Neel Nanda (Sep 2025) ⭐️\nSo you want to work on technical AI safety by George Wang (Jun 2024) ⭐️\nAI safety technical research by Benjamin Hilton (Jun 2023) (LessWrong version)\nAGI safety career advice by Richard Ngo (May 2023) ⭐️\nLevelling Up in AI Safety Research Engineering by GabeM (Sep 2022)\nHow to pursue a career in technical AI alignment by Charlie Rogers-Smith (Jun 2022) ⭐️\nFAQ: Advice for AI alignment researchers by Rohin Shah (Jan 2021)\nBelow, I\u0026rsquo;ll highlight several concrete principles for someone like me to get started.\nMentorship This is the first thing that every person and career advice blog I came across mentioned.\nIn Richard Ngo\u0026rsquo;s AGI safety career advice, he specifically emphasizes:\nIf you have little research experience, your main priority should be finding the best mentor possible to help you gain research skills\nEthan Perez echoes this in Tips for Empirical Alignment Research:\nClose mentorship is maybe the fastest path to become an expert in a domain, and there\u0026rsquo;s a lot of evidence to support this.\nThe data backs this up: In MATS\u0026rsquo; alumni evaluation, 72% said mentorship was either the most valuable aspect or represented a major share of the program\u0026rsquo;s value.\nHere\u0026rsquo;s why I think mentorship is considered so critically important in technical alignment research:\nFast feedback loops on your work and thinking Network connections through your mentor\u0026rsquo;s professional relationships Higher publication success rates through guided research Proven track record - mentoring correlates with better career and productivity outcomes across many research domains, not just alignment The central importance of mentorship leads to one of the key challenges in entering technical alignment research: Alignment is mentorship-constrained. The field remains small, with far fewer experienced researchers than aspiring newcomers.\nHere are a few ways to secure mentorship without quitting your job:\nPart-time research programs (e.g., SPAR, MARS, etc. - see the research programs section) Direct outreach to professors or PhD students working on AI safety to see if you can contribute to their papers Have Something to Show This is another universal recommendation across all the career advice I\u0026rsquo;ve encountered.\nGeorge Wang (research lead at Timaeus) explains the incremental approach in So you want to work on technical AI safety:\nThe general idea here is to do a small thing to show that you\u0026rsquo;re a good candidate for a medium thing, then do a medium thing to show you can do a bigger thing, and so on.\nYou need to give interviewers, mentors, or professors a concrete reason to work with you. A portfolio serves as a strong signal to prevent false positives—hiring someone they shouldn\u0026rsquo;t have hired.\nNeel Nanda emphasizes this point in How To Become A Mechanistic Interpretability Researcher:\nFurther, if you want to pursue a career in the space, whether a job, a PhD, or just informally working with mentors, public research output is your best credential. It\u0026rsquo;s very clear and concrete proof that you are competent, can execute on research and do interesting things, and this is exactly the kind of evidence people care about seeing if they\u0026rsquo;re trying to figure out whether they should work with you, pay attention to what you\u0026rsquo;re saying, etc.\nWhat hiring managers ultimately look for:\nYour goal as a candidate is to provide compelling, hard-to-fake evidence of your skills. The best way to do that is to simply do good research and share it publicly.\nSkip Forward Aggressively George Wang advocates for an accelerated approach in So you want to work on technical AI safety:\nAs a general note, I don\u0026rsquo;t recommend that you try to do everything listed in depth. I\u0026rsquo;m trying not to leave huge gaps here, but you can and should try to skip forward aggressively, and you\u0026rsquo;ll probably find that you\u0026rsquo;re ready for later steps much sooner than you think you are.\nLet\u0026rsquo;s say the entire learning roadmap looks like this: I should attempt step n+1 or even n+2 while I\u0026rsquo;m still working on step n. Sometimes I might already be capable of performing step n+1 despite still being at step n. Other times, I\u0026rsquo;ll predictably hit a wall or blocker when trying to reach ahead.\nBut hitting that wall is actually valuable—now I know exactly what to work on next. That\u0026rsquo;s progress in itself. If I hadn\u0026rsquo;t tried applying, I might still be over-studying unnecessary material or remain unaware of the actual requirements.\nI believe the general principle is to leap forward quickly, especially given that AI safety timelines are short. I don\u0026rsquo;t need to become a top researcher to land a research role—I just need to meet the minimum bar and learn the rest on the job.\nThis approach is validated by real hiring practices. In a blog post about hiring for safety researchers, the Google DeepMind team stated:\nAs a rough test for the Research Engineer role, if you can reproduce a typical ML paper in a few hundred hours and your interests align with ours, we\u0026rsquo;re probably interested in interviewing you.\nExcel at Your Current Job Excelling in your current role builds essential career capital—the skills, reputation, professional network, and financial runway that top alignment teams actively seek when hiring.\nEqually important: strong software engineering skills are non-negotiable for technical alignment positions. Frontier research labs predominantly hire \u0026ldquo;research engineers\u0026rdquo; who design experiments, build specialized tooling, and manage large distributed systems. These requirements align perfectly with the experience gained from high-performing software engineering roles.\nExample: Redwood Research careers page Resources Resource hubs Events \u0026amp; training by AISafety.com ⭐️\nAGI careers by 80,000 Hours (they also offer 1-1 career advising)\nAI Safety Papers \u0026amp; Opportunities by Arkose\nOrganizations Map of AI safety orgs by AISafety.com ⭐️ - the most comprehensive landscape for ai safety related orgs\nResearch programs MATS (in-person, full-time, Berkeley/London)\nSPAR (part-time)\nMARS (part-time)\nAstra Fellowship by Constellation (in-person, full-time, Berkeley)\n","permalink":"https://eliwangj.github.io/posts/ai-safety-career-exploration/","summary":"Sometimes you can just do things.","title":"Finding My Way Into AI Safety"}]