Hi Tristan,
Merry Christmas and Happy Holidays! It’s been a wild year for both AI Safety specifically and long-termism or existential risk reduction in general. Between the release of GPT-4, various governmental proposals and policies to regulate frontier models, and increasing recognition by prominent experts and policymakers of the need to pay attention to near-term existential risks, it’s been a wild time. Not to mention the ongoing hot war in Russia/Ukraine and the lingering effects of COVID…
Amidst all the excitement and chaos of the last year, I’m really grateful for the 4,000 people who’ve donated to the Long-Term Future Fund (LTFF). Thank you for donating! Your donations have allowed us to award many excellent grants to impactful projects and individuals. It is truly inspiring to see so many people give their hard-earned money to projects they genuinely believe are making the most positive impact, from college students giving $5/month to dedicated earning-to-givers who donate over 50% of their income.
This is the inaugural (hopefully annual) newsletter for the Long-Term Future Fund, where we update you on the most important goings-on for LTFF in the last year. We sent you this newsletter either because you subscribed or because you’ve donated to LTFF over the last two years. To unsubscribe, click here or see the bottom of this email.
Click here (every.org), if you wish to donate to LTFF before 2024.
Major updatesGrants made so far: LTFF has given $6.67M across 197 grants since the start of the year. We are aiming for an ideal dispersal rate of $700,000/month for 2024 ($8.4M/year). We currently have about 6 months of funding in reserve. OP and LTFF distancing: To increase funder diversity and independence, LTFF and Open Philanthropy are distancing themselves from each other. This means that we will rely more on individual donors, such as yourself. Donation matching: As part of the distancing, Open Phil offered 3.5M in matching funds to donations to the LTFF. Leaving Effective Ventures. Along with other organizations, we plan to leave our fiscal sponsor (Effective Ventures). We do not have a timeline for this, but will keep you posted. We do not expect the change to be a significant inconvenience for either our donors or grantees. Personnel updates: Also as part of the aforementioned distancing, Asya Bergal (who joined Open Phil while working part-time at LTFF) has formally stepped down from her role as LTFF fund chair. Caleb Parikh (head of EA Funds) is serving as interim chair, and we’re currently in the process of hiring a new LTFF fund chair. Caleb Parikh, Oliver Habryka, and I are the permanent fund managers. This year, we have also brought aboard Lawrence Chan, Clara Collier, Lauro Langosco, and Daniel Eth (returning), who will be joining Thomas Larsen as guest fund managers. See more Grants database: Though technically launched last year, we now have a grants database for all of the non-confidential grants we’ve given out. I (Linch Zhang) joined EA Funds full-time as the second full-time employee, leaving my role as Senior Researcher at Rethink Priorities. I was previously a part-time fund manager (grantmaker) at LTFF. I joined EA Funds because I think now is a critical period to improve EA funding. I will primarily work on the Funds’ public communication (hi!), but will also help out with grant evaluations, fundraising, strategy setting, and hiring. I’m working with EA Funds and several other people to launch the AI Risk Mitigation Fund, a new fund dedicated solely to AI safety and reducing AI catastrophic risk. The idea is to create an excellent donation option for people who want to avert AI catastrophic risk, but might not be EA or longtermist or share the rest of (e.g.) LTFF’s priorities, like biosecurity, forecasting, or longtermist philosophy. We want the soft launch to be lowkey for now, no public fanfare but a viable end-of-year donation option for people focused on AI risk. Check out the website here, see the soft launch post here, or donate here.
Highlighted Grants from the last two yearsLogan Smith ($40,000) - 6-month stipend to create language model (LM) tools to aid alignment research through feedback and content generation Noemi Dreksler ($231,800) - Two-year funding to conduct public and expert surveys on AI governance and forecasting The project was aimed at expanding the world’s understanding of public and expert opinions on AI, through survey design and analysis. Dreksler aimed to foster informed discussions in academic and policy circles about AI's long-term impact. This is crucial for developing informed AI governance strategies. Dreksler has a DPhil in Experimental Psychology and experience in AI-focused survey research at the Centre for the Governance of AI. She is collaborating with various established researchers including Baobao Zhang, Allan Dafoe, and Markus Anderljung. Dreksler and collaborators used this grant to publish a number of important surveys, which help inform policymaking and further technical research:
Lennart Heim ($67,800) - 7-month stipend to conduct a research project collaboration on compute governance with Markus Anderljung from GovAI At the time that he applied (November 2021), Heim was an early-career researcher who had written a blog sequence on AI and compute during a summer internship in AI governance. He had also done some work on hardware engineering and machine learning. Since we made the grant, Heim has become a recognized expert in AI governance and policy through several positions and contributions:
MATS program ($316,000) - 8-week scholars program to pair promising alignment researchers with renowned mentors. In 2022, we gave to the second iteration of SERI MATS (now just MATS), a training program for new alignment researchers. This program has now grown into a more established program producing multiple people working full-time on alignment in established research organizations (with a smaller number of people pursuing independent research or starting new organizations). MATS is now primarily funded by larger sources of institutional funding like Open Philanthropy.
AI X-risk Podcast ($24,000) - Support for 12 episodes of the AI X-risk Podcast (AXRP) We supported Daniel Filan’s AI X-risk podcast for $24,000 for 12 episodes. We have been the first to provide significant financial support for his podcast. Some ML researchers have told us it is their favorite podcast on AI safety, and we think it is among the best technical introductions to current research on AI safety.
Joseph Bloom ($50,000) - 6-month stipend to conduct AI alignment research on circuits in decision transformers We supported Joseph Bloom (LessWrong) to conduct AI alignment research into circuits in decision transformers. Some fund managers were excited about initial results and thought Joseph had a very solid and useful writeup.
Philip Tetlock, Ezra Karger, Pavel Atanasov ($200,000) - Existential risk forecasting tournaments We funded a major AI x-risk forecasting tournament with superforecasters, for a project that would later become the Forecasting Research Institute. Tetlock is of course one of the most famous research scientists studying forecasting. We thought the tournament would be helpful partially for grounding out useful numbers for further decision-making purposes, but mostly for drawing attention to the dangers of AI x-risk and also providing a common language for people outside of our narrow community to talk about AI x-risk. We also assisted them in obtaining additional funding elsewhere.
Jeffrey Ladish ($98,000) - 6-month stipend & operational expenses to start a cybersecurity & alignment risk assessment org We funded Jeffrey Ladish, formerly an early information security engineer at Anthropic, to create an organization to assess and communicate risks from AI systems, with a focus on cybersecurity and alignment risks. This organization later became Palisade Research. I’ve heard good things about Palisade from people in adjacent fields (including other LTFF fund managers), but I (Linch) confess I have not personally kept up with the relevant details.
Anonymous ($60,000) - Media explaining developments within AI and their impact on society In 2023, we gave a grant to a very prominent technical content creator to pivot more of their work to discussing AI safety, as well as the overall long-term impacts of AI on society. We were excited by their reach, quality of audience, and the level of accuracy and care they put into their content. They asked to be anonymous.
Sage Bergerson ($2,500) - 5-month, part-time stipend for collaborating on a research paper analyzing the implications of compute access with Epoch, FutureTech (MIT CSAIL), and GovAI Sage Bergerson sought funding to support research analyzing the implications of compute access, working with the research nonprofit Epoch. The research paper Sage Bergerson is collaborating on aimed to analyze the implications of compute access, focusing on how research agendas in academic and industry machine learning labs are evolving due to this access. It also sought to address the risks associated with decreased scrutiny of machine learning models in the industry, providing evidence informative to those working in AI policy. This project aimed to influence strategic policy decisions to integrate machine learning into society more smoothly and safely, reducing uncertainties about the future of AI. She has completed an NLP research fellowship project at NYU and wrote a paper the summer before as a fellow at the Stanford Existential Risk Initiative (SERI). During the grant, she also interned at the Office of the Director of National Intelligence within the US Government.
You can also read more about our grants in the most recent payout report. However, note that the report does not have grants given out past April 2023.
All of our public writings this yearLTFF mostly does not have an institutional “voice” or institutional writing. Nonetheless, below are writings from LTFF fund managers that donors and other community members may find helpful in understanding how LTFF fund managers think about our grantmaking, and/or other related issues.
LTFF April 2023 grant recommendations: Our omnibus payout report, to help donors and others get a relatively detailed sense of what we’ve been funding and why. What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund: A detailed discussion of “grantmaking thresholds” for marginal grants at LTFF. Essentially, given that we have limited resources and many good projects to fund, how do we choose which grants to make per $X we have? The post covers different projects we might want to fund at different thresholds ($X per 6 months). Select examples of adverse selection in longtermist grantmaking: I reviewed my past experiences with “adverse selection” as a grantmaker, that is, situations where we choose to not fund a project that initially looked good, often due to surprising and private information. LTFF and EAIF are unusually funding-constrained right now: Our fundraising post in September. We are in much less of a funding crunch now than we were in September, but the post may still be helpful for you to decide whether LTFF (or EAIF) are good donation targets relative to your next best alternative. The Long-Term Future Fund is looking for a full-time fund chair: Our hiring post for LTFF fund chair. Mostly a historical curiosity now that we’re no longer looking at new applications, but community members may be interested in reading it to understand the responsibilities and day-to-day of work at LTFF. Hypothetical grants that the Long-Term Future Fund narrowly rejected: A continuation of the marginal grants post, in that it’s a more narrow and tightly scoped list of hypothetical grants that are very close to our current funding bar. If you’re considering whether to fund LTFF or not, I think this post may be the best one in helping you decide what the most likely uses of your marginal dollars would end up actually funding. LessWrong comments discussion of whether longtermism or LTFF work has been net negative so far: A LW comments discussion of whether we should be worried about donating to LTFF, asking for assurances that LTFF will not fund net-negative work. Multiple fund managers offered their individual perspectives. Tl;dr: We are unfortunately unable to provide strong assurances. :/ Doing robustly good work in a highly speculative domain is very difficult, and fund managers are not confident that we can always be sure our work is good. Lawrence Chan: What I would do if I wasn’t at ARC Evals: Lawrence Chan, a part-time guest fund manager at LTFF, discusses what he’d likely do if he wasn’t at ARC Evals (his day job). This might be relevant to community members considering career pivots in or into AI Safety/x-risk reduction. Caleb Parikh on AI consciousness: Caleb Parikh, Project Lead of EA Funds and interim LTFF fund chair, discusses why he thinks the broader community is underinvesting in research projects working on AI consciousness.
AI Risk Mitigation Fund (quick aside)I and some other people are soft launching the AI Risk Mitigation Fund (ARM Fund) in time for last minute end-of-year giving. The fund has three core focus areas: technical AI safety research, the establishment of beneficial AI policies, and the cultivation of new talent in AI safety research and governance.
The idea is that rapid AI progress has probably made many people reconsider their views on catastrophic risks from AI. AI safety is more in the public conversation than it has ever been. But there isn’t really an excellent public donation option for people solely concerned about AI Risk. They might not (e.g.) really buy the case for existential biosecurity, forecasting, longtermism, etc. So we want to create a fund that serves the philanthropic needs of people in that demographic.
If you think you might know people in that camp (or you consider yourself to be in that camp), consider sharing the website with them! Donation link here.
This is not an LTFF project, but it shares many personnel and infrastructure with LTFF for now. We want the brand to be quite distinct (Again, you don’t need to be EA or longtermist, etc, to think that reducing AI risk is a good idea). If things go well and the ARM Fund becomes sufficiently popular and well-funded, we’ll also want greater practical separation in the near future (e.g. very distinct personnel, likely different grant selection process and funding bars, maybe spin out as a distinct nonprofit, etc).
We’re launching in the last week of December to give unusually eager and/or procrastinating donors a chance to donate to the new fund before the US tax year ends. Because we aren’t 100% ready, we want to keep the soft launch low-key for now and just spread the website and knowledge of the fund through word-of-mouth and emails. (We’ll do a fuller launch early next year).
More details in the launch post here (website, donation link). What is happening?! What is happening in the world? For donors who are very busy and/or understandably haven’t kept up as much with the news, I’ve compiled a list of interesting and important longtermism- / x-risk- / AI Safety- relevant news. (For people who have been keeping up, feel free to skip this section!)
There’s been a lot of dramatic (and to some of us, scary) progress towards AGI via large language models, GPT-4 release on March 14 being the most prominent example. To get a sense of just how quickly things have been progressing, consider engaging with this interactive explainer. In April, Alphabet’s two AI research divisions, DeepMind and Google Brain, merged to become Google Deepmind, headed by DeepMind CEO Demis Hassabis. Technical AI Safety advances: (Note that I relied on advisors for this section, as I am not an expert in ML myself and feel underqualified to give a survey of the field) There’s been a lot of advances on using sparse autoencoders to solve the problem of “polysemanticity” Polysemanticity is when a neuron responds to multiple, unrelated features. This makes it very hard to interpret the internals of ML models, an important element for various alignment and safety strategies. In 2023, people have been able to use sparse autoencoders to reliably extract and identify a model’s monosemantic features, individual features corresponding to human-identifiable concepts. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning is Anthropic’s paper on this. Anthropic’s paper follows concurrent work (presented in ) by several independent researchers, including LTFF grantees.
AI Control: Improving Safety Despite Intentional Subversion by Greenblatt et.al from Redwood Research, designs and tests safety techniques that prevent models from causing bad outcomes even if the models collude to subvert them. The problem setup assumes a) maybe-misaligned powerful models, b) aligned-but-less-powerful models, and c) a small amount of human labor. The authors believe that this is a realistic model of when catastrophic AI safety concerns will arise in the near-medium future. The core idea is to ensure safe outcomes even with misaligned models. They test various approaches using finetuned misaligned GPT-4 as the maybe-misaligned model, and regular GPT3.5 as the safe model. They developed two defenses that seem to work well in this problem setting.
Some advisors also recommended the ML Safety newsletter for a roundup of top safety papers of 2023. 2023 might go down in history as THE year where academics, public intellectuals, and policymakers increasingly woke up to, and openly concerned themselves about, AI existential risk: Open letters of concern about AI risk are being signed by very prominent people: FLI letter from March asking for a 6-month pause in training runs of “Giant AI Experiments,” that is, very large LLMs. Signed by 33,000+ people, including prominent researchers, Elon Musk, and Andrew Yang. AI Treaty letter signed by prominent Chinese and Western AI academics, urging for an international treaty with teeth on AI Safety. And perhaps most importantly, the Center for AI Safety’s Statement on AI Risk. Worded extremely simply, it has just one line: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” This letter has been signed by Geoffrey Hinton and Yoshua Bengio, two Turing Award winners for their work on ML, as well as the respective CEOs of Google DeepMind, OpenAI, and Anthropic, as well as many other very prominent AI researchers.
The “first global AI summit,” on AI Safety, was held in November in the UK! The Summit culminated in participating countries, including the US, China, and the EU, signing the Bletchley Declaration, which called for international cooperation in managing risks from AI and specifically called out “unintended issues of control relating to alignment with human intent.” Follow-up summits are expected, with South Korea and France being expected to host the next two summits over the next year.
White House Issues (popular) Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence The EO Mandates reporting requirements for developers of large foundation models to “share their safety test results and other critical information with the U.S. government.” Asks the National Institute of Standards and Technology (NIST) to develop “rigorous standards for extensive red-team testing to ensure safety before public release,” and for the Departments of Energy and Homeland Security to apply those standards. Has a specific provision for guarding against risks of using AI/LLMS for generating dangerous biological capabilities.
In initial polling: Overwhelming support for the executive order Highest support specifically for the parts of the EO that are most x-risk relevant. Most people thought the EO didn't go far enough.
EU AI Act: first regulation on artificial intelligence As a sign for how the times have changed, only eight months ago, Fox News White House correspondent Peter Doocey asked the White House spokesperson about the dangers of advancing AI when there’s a risk that “literally everyone on Earth will die.” At the time, everybody in the room laughed. These days, I think most people no longer consider it a laughing matter. OpenAI launched their “Superalignment” team (July), and the team now gives out Fast Grants (Dec). Many people are now talking about the near-term potential for LLMs to spark the next man-made pandemic. For example, the White House EO (above), Rishi Sunak here, MIT Prof Kevin Esvelt here, and Esvelt lab’s initial research here. Personally I’m kind of confused overall, see also this critique on LessWrong. It’s easy to forget in the comfort of the West Coast, but the last two years have probably been the riskiest the world has seen for nuclear war since the Cold War. I enjoyed this conceptual paper presented by Dan Hendrycks, [2303.16200] Natural Selection Favors AIs over Humans, which argues that evolution by natural selection a) selects for selfishness, and b) natural selection may be a dominant force in AI development.
What I’ve been reading and ponderingIn future newsletters, I’d like to get inputs from other fund managers, grantees, donors, and advisors involved in LTFF for this section. But for now, we just have my own self-indulgent takes. At EA Funds, we’ve been reading How To Measure Anything. I’m halfway through, and I quite like the book! It has many practical tips on the when, why, and how to easily quantify uncertainty.
I enjoyed skimming Can We Survive Technology, a 1955 essay by John von Neumann that grapples with many issues that people in our circles think about today. Recent events have led me to viscerally consider the hypothesis that the future may get increasingly crazy for multiple years before the singularity, especially in slow-AI takeoff worlds. Intuitively it feels like there should be stuff I can do to prepare for “crazy times,” but I don’t actually have a good sense of how (a different fund manager recommends Habryka et.al’s dialogue on ethical AI investments). In terms of fiction, I enjoy rereading Isaac Asimov’s Foundation series. At the high level, the series is about preparing for the collapse of civilization, and ensuring a speedy recovery, which of course at least rhymes with some of my LTFF-related interests. I also enjoyed reading Beyond the Burn-Line, a postapocalyptic, posthuman science-fiction novel that explored several long termism-relevant topics The “Shoggoth with a smiley face” meme has taken over the internet by storm, as a metaphor for modern AI.
Thomas Andrews was the lead designer for the Titanic. He wanted many more safety features, but “Andrews's suggestions that the ship have 48 lifeboats (instead of the 20 it ultimately carried) as well as a double hull and watertight bulkheads that went up to B deck, were overruled.” After the Titanic crashed, he died trying to rescue and evacuate passengers.
Consider donating to us!I’d be remiss to not ask you to donate to us again. 😛
We’d love to have your continued support, assuming that you continue to believe that we are the best use of your donations. I think marginal donations to the Long-Term Future Fund can be very impactful, even past our current target of $700,000/month (up to around $1M/month). Here are some (hypothetical/anonymized) examples of grants that are very close to our funding bar, but we couldn't currently fund due to insufficient resources: ($25,000) Funding to continue research on finding internal search algorithms within a multi-modal chess language model, focusing on alignment and interpretability. ($25,000) Four months' stipend for a former academic to tackle some unusually tractable research problems in disaster resilience after large-scale GCRs. ($50,000) Six months of career transition funding to help the applicant enter a technical AI safety role. ($100,000) 16 months of funding for a PhD completion, focusing on partially observable reward learning and developmental interpretability. ($7,000) Four months of funding for research on enhancing AI models' ability to learn and interpret social norms, leading to the preparation of an academic paper submission.
More details in the link.
If you consider the projects we fund to be impactful, and in particular, more impactful on the margin than the other donation options you’ve looked at, please consider donating to us.
If someone you know is interested in doing direct work in improving the long-term future, please consider asking them to apply for funding from us. We ask people to have a pretty low bar for applying, as we do not want to deter otherwise great applicants from applying to us. Even though we’re more funding constrained than before, more applications should still increase the average quality of our grants.
Thank you so much for your time! This is my first LTFF newsletter, so any feedback is highly appreciated!
Merry Christmas and Happy New Year!
Linch |
