{"id":6277,"date":"2024-04-07T05:08:30","date_gmt":"2024-04-07T12:08:30","guid":{"rendered":"https:\/\/athis-technologies.com\/news\/?p=6277"},"modified":"2024-05-27T05:18:12","modified_gmt":"2024-05-27T12:18:12","slug":"ai-review-2024","status":"publish","type":"post","link":"https:\/\/athis-technologies.com\/news\/market\/2024\/ai-review-2024\/","title":{"rendered":"AI review 2024"},"content":{"rendered":"\n

\ud83c\udf0e The (geo)politics of AI<\/strong><\/em><\/h3>\n\n\n\n

The AI world was stunned by the surprise appointment<\/a> of Inflection co-founder Mustafa Suleyman as CEO of Microsoft AI. The newly-created division will focus on the company\u2019s consumer AI efforts, with Copilot as its flagship product. Mustafa will bring a number of key Inflection staffers with him, while Microsoft will pay<\/a> his former company $650M in licensing fees to provide access to Pi via Azure – presumably as a form of pseudo-acquisition payout to shareholders. Meanwhile, Inflection has said<\/a> it will pivot away from its consumer-facing chatbot in favor of enterprise services.<\/p>\n\n\n\n

To call the sequence of events here unusual is an understatement, but we think it\u2019s a sign of things to come. In the past year, we\u2019ve seen the performance of foundation models converge, while already sky-high GPU bills continue to grow. Usually a market in which a group of companies are building similar products but are struggling to make the economics work would see consolidation. However, regulators on both sides of the Atlantic have made it clear that they don’t want this to happen.<\/p>\n\n\n\n

Freezing the exit market might preserve the illusion of competition, but it doesn\u2019t make non-viable businesses any more viable. As this story shows, it ends up punishing people for taking risk, whether that\u2019s investors or early employees, who both stand to lose big. Meanwhile, we\u2019ll be watching Mustafa\u2019s next act closely. He may be divisive<\/a>, but he\u2019s never boring\u2026<\/p>\n\n\n\n

While we\u2019re on the subject of competition regulators, we can\u2019t see them being thrilled by the rumors that Apple plans to integrate Google\u2019s Gemini into the iPhone. Apple has kept its AI cards close to its chest over the past few months, although we do know they have various on-device projects cooking behind the scenes. This silence, combined with recent internal resource reallocation<\/a>, has led to speculation that these efforts aren\u2019t running entirely smoothly, although there appear to be signs of change. More on that in the research section\u2026 <\/p>\n\n\n\n

We don\u2019t know what this partnership would involve, but it\u2019s easy to imagine Siri getting a new lease of life or Apple moving to supercharge its image and video-editing capabilities. Either way, it would represent a huge boost for Gemini and serve as a powerful reminder not to write Google off in the AI race, despite some of its teething problems. That said, this partnership will undoubtedly be received poorly in some quarters. Google\u2019s 2021 deal with Apple to make its Search the default search engine on Safari is now at the center of a landmark US antitrust action<\/a>. Throw in Google\u2019s plans<\/a> for a paid AI-powered search engine and it\u2019s hard to imagine the EU and the FTC staying away for long.<\/p>\n\n\n\n

Moving from anti-trust to safety, back in February, the US National Telecommunications and Information Administration (NTIA), an arm of the US Department of Commerce, put out a Request For Comment<\/a> on the risks and benefits of open AI models with widely available weights. Since then, we\u2019ve seen responses pour in from a range of tech companies, academic institutions, and think tanks. While these vary quite dramatically in detail level, they provide a helpful illustration of the emerging battlelines in this debate. <\/p>\n\n\n\n

Unsurprisingly, Meta<\/a> put up a robust defense of openness along with Stanford HAI<\/a>, while OpenAI<\/a> studiously avoided any specifics, and Google DeepMind<\/a> tried to steer a middle path, reframing the debate away from open vs closed. The Center for AI Policy<\/a> invoked the prospect of \u201cWDM-like AI\u201d in a response that called for more immediate regulation. But for us, many of these responses are more interesting for what they don\u2019t say. <\/p>\n\n\n\n

With the exception of Meta, the main labs all implicitly believe that there will have to be some form of foundation model regulation in the future. However, they don\u2019t wish to be pinned down on a specific capability threshold or commit to a theory of liability for model misuse before this happens. This inevitably results in artistic levels of vagueness in some of their policy recommendations. Problems are surfaced, before an amorphous mix of government, civil society, industry is tasked with figuring out the solutions. This triggers flashbacks to the blank cheque<\/a> the then Facebook gave policymakers to regulate it at the height of the backlash against big tech. They lived to regret it and learned their lesson. Will other companies find themselves in the same position in a few years time?<\/p>\n\n\n\n

While the labs agonize over the wording of their recommendations, the UK and US Governments are moving forward with the commitments made at last November\u2019s AI Safety Summit. The UK and US unveiled an MOU<\/a>, which will see the two countries\u2019 respective AI safety institutes align their standards for testing and evaluating powerful AI models. Despite the fanfare, the MOU contains little we didn\u2019t know before. Our more cynical readers might argue that this is the product of a government scrabbling around for good news or relevance. We couldn\u2019t possibly comment.<\/p>\n\n\n\n

\ud83c\udf6a Hardware<\/strong><\/em><\/h3>\n\n\n\n

The past month saw NVIDIA\u2019s GTC and an accompanying run of announcements. Top of the list for most people was the unveiling of the new Blackwell<\/a> B200 GPU and GB200 Superchip. The system will mark a significant performance gain on the previous Hopper architecture of H100 fame, with the company claiming it can reduce cost and energy consumption 25x over an H100. To illustrate this, the company said that it was possible to run a 1.8 trillion parameter model on 2,000 Blackwell GPUs while consuming 4 megawatts of power versus 8,000 Hopper GPUs and 15 megawatts. Full production and shipping should begin later this year, but NVIDIA hasn\u2019t committed to a specific timeframe. Strikingly, and in an illustration of NVIDIA\u2019s power, seemingly every big AI lab CEO pledged their allegiance in the accompanying press release. As the saying goes, when asked to jump, the answer is \u201chow high?\u201d.<\/p>\n\n\n\n

To much less fanfare, Cerebras unveiled<\/a> the CS-3, underpinned by the new WSE-3 chip, which they are billing as the fastest AI accelerator in the world. The company is claiming that the CS-3 makes it possible to train a 24 trillion parameter model (over 10x bigger than the current largest frontier models) on a single device. Cerebras appears to be planning<\/a> for an IPO in the second half of this year, targeting an increase on its current $4 billion valuation. Considering the company has struggled to compete with NVIDIA at scale, due to its admittedly powerful product being difficult to manufacture and scale economically, an IPO may seem like a bold move. But as the rising public market tides continue to lift all semiconductor boats, it may well pay off.<\/p>\n\n\n\n

Past editions of the newsletter have covered Sam Altman\u2019s ambitions for a multi-trillion dollar international chip empire that breaks OpenAI\u2019s dependence on NVIDIA. OpenAI is rumored<\/a> to be in talks with Abu Dhabi state-backed investor MGX, following separate discussions<\/a> with Singapore-backed Temasek. There are separate rumors that OpenAI has been pitching<\/a> Microsoft on plans for a $100 billion super monster enormous data center. If approved, this would be orders of magnitude larger than any other Microsoft data center. It would also be set up to give Microsoft and OpenAI the flexibility to use alternatives to NVIDIA GPUs.<\/p>\n\n\n\n

This all reinforces the tightrope that OpenAI is (so far quite masterfully) walking. On the one hand, the eye-watering sums of money its ambitions require are far beyond the reaches of standard VC funding, but international sovereign wealth is fraught with risk. If we take MGX as an example, one of its founding partners<\/a> is G42, which attracted concerns from US intelligence officials over its ties to China<\/a>. The UAE and China have been forging closer military ties<\/a>, while Chinese-developed surveillance technology is deployed <\/a>on the streets of its cities. While these issues are removed from chips per say, it\u2019s not hard to envisage the Congressional subcommittees that previously investigated TikTok taking an interest. For now, OpenAI just needs GPT-5 to wow and for Microsoft to keep writing the checks. No pressure then.<\/p>\n\n\n\n

Europe meanwhile might be able to gain a surprising new semiconductor hub: Italy. After the country failed to bribe<\/s> convince Intel enough to build out local capacity, Singapore-based semiconductor integrator Silicon Box will invest <\/a>\u20ac3.2 billion to build a chip factory in northern Italy. This deal will likely have received significant government financial support, but the details are yet to be published. As with many of these European chip subsidy arrangements, we remain skeptical that onshoring tiny portions of the supply chain guarantees \u2018sovereignty\u2019 in the event of an actual geopolitical shock. That said, this all pales in comparison to the $20 billion in grants and loans Intel will receive<\/a> from the US government via the Chips Act.<\/p>\n\n\n\n

\ud83c\udfed Big tech<\/s> start-ups <\/strong><\/em><\/h3>\n\n\n\n

As the open source ecosystem continues to go from strength to strength, Databricks has emerged a surprise new leader with the release of DBRX<\/a>. The model uses a fine-grained mixture-of-experts (MoE) architecture, encompassing 132B parameters. It surpasses competitors like Mixtral and Grok-1 in granularity by operating with 16 smaller experts – selecting 4 for any given task – facilitating a significantly higher diversity in expert combinations that enhance model performance. The model was pretrained on 12T tokens of curated data and a maximum context length of 32k tokens, with the team finding that curriculum learning for pre-training drove significant improvements in results. This is interesting validation for a technique that\u2019s long had potential, but has been under-discussed as an approach to improving training performance. <\/p>\n\n\n\n

The resulting model outperformed established open source models across a range of benchmarks, while surpassing GPT-3.5 and being competitive with Gemini 1.0 Pro.<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

The work took the team approximately 3 months, a relatively lean $10M<\/a>, and required 3,072 NVIDIA H100s. However, it\u2019s always worth taking the cost figures from tech companies with a pinch of salt, as they don\u2019t include investments carried over from previous models or projects. <\/p>\n\n\n\n

After Elon Musk launched his legal action against OpenAI for its lack of openness, a number of AI commentators on X questioned why Grok was closed. In response, xAI released<\/a> the base model weights and network architecture of Grok-1. Openness is welcome but, given the size of the model, most people would struggle to run it locally without access to a GPU server. In a busy few weeks for the company, xAI also announced<\/a> Grok-1.5. As well as a new 128k context length, xAI recorded significant performance gains in coding and math-related tasks, with a particularly strong performance on HumanEval, a code generation and problem-solving benchmark. The model is competitive with Mistral Large, Gemini Pro, and the March 2023 release of GPT-4.<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

After impressing observers with the new Claude 3 family, Anthropic received<\/a> $2.75B from Amazon as part of its $4B commitment to the company. As part of the arrangement, Amazon will be the company\u2019s primary (although not exclusive) provider of cloud services. Anthropic is still conducting a separate fundraising process, expected to total<\/a> at least $750M. One early Anthropic investor we don\u2019t hear much from these days is FTX. The majority of the fallen crypto exchange\u2019s 8% stake in the company was sold this month<\/a>, with the largest buyer being an affiliate of Emirati sovereign wealth fund Mubadala. A bid from Saudi Arabia was turned down on national security grounds.<\/p>\n\n\n\n

OpenAI has revealed<\/a> that it has been building Voice Engine, a text-to-speech and voice cloning model. The company first developed it in 2022 and used it to power some of the text-to-speech functionalities bundled with ChatGPT. The company has now released snippets of audio in multiple different languages, but has opted not to make the model accessible, citing safety concerns. OpenAI warns that we urgently need to improve public awareness of the potential of deceptive content, phase out voice-based authentication as a security method, and build better provenance tools. While some immediately accused<\/a> OpenAI of using safety fears as a marketing tactic, considering the number of major elections looming this year, it\u2019s understandable that the company may have decided that this isn\u2019t a regulatory storm worth sailing into just yet.<\/p>\n\n\n\n

\ud83d\udd2cResearch<\/strong><\/h3>\n\n\n\n

A generalist AI agent for 3D environments<\/a>, Google DeepMind<\/em>.<\/p>\n\n\n\n

Presents SIMA, a generalist agent that can follow natural language instructions to navigate and interact within a wide range of 3D virtual environments, including research platforms and video games, marking a significant advance on systems trained for specific environments. <\/p>\n\n\n\n

The core of SIMA\u2019s architecture is an image-language model, combined with a video prediction model that anticipates how the environment will change over time. The DeepMind team partnered with 8 game studios to expose SIMA to 9 different video games, ranging from open-world exploration tophysics-based sandboxes. They also used 4 custom research environments, including a Unity-based “Construction Lab” where agents have to build sculptures from building blocks. This diversity of 3D environments was crucial to imbuing SIMA with broad, generalizable capabilities. The researchers further trained SIMA on a large dataset of human-generated language instructions paired with the corresponding game actions, recorded from gameplay sessions where one player provided instructions to another.<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

The multimodal approach allowed SIMA to interpret instructions and execute actions in unfamiliar settings, demonstrating robust zero-shot learning abilities. Based on an evaluation of 1,500 unique in-game tasks, the researchers found that in each game, SIMA outperformed all the specialized agents.<\/p>\n\n\n\n

Long-form Factuality in Large Language Models<\/a>, Google DeepMind<\/em>.<\/p>\n\n\n\n

Introduces a new framework, consisting of a dataset, evaluation tool, and metric, for assessing the factuality of LLMs in producing long-form content.<\/p>\n\n\n\n

LongFact is a dataset consisting of 2,280 fact-seeking prompts (generated by GPT-4) across 38 topics, designed to assess the ability of LLMs to generate accurate long-form responses. SAFE, the evaluation tool, automates the process of dissecting a model’s response into individual facts and assessing their accuracy using Google Search in a multi-step reasoning process. This approach is demonstrated to be more reliable and over 20 times cheaper than human annotators.<\/p>\n\n\n\n

The evaluation metric, F1<\/sub>@K, is an adaptation of the F1<\/sub> score that balances the precision and recall of a response, taking into account the number of supported facts relative to a hyperparameter representing a user’s preferred response length (K).<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

The results from benchmarking thirteen models from across four families (Gemini, GPT, Claude, and PaLM-2) found that larger models within these families performed better in terms of long-form factuality. Between families, GPT-4 Turbo was the strongest performer, followed by Gemini-Ultra, and Claude-3 Opus.<\/p>\n\n\n\n

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training<\/a>, Apple<\/em>.<\/p>\n\n\n\n

Provides a glimpse into Apple\u2019s approach into developing high-performing large multimodal models. The authors lay out their findings on optimal design, finding that using a mix of image-caption, interleaved image-text, and text-only data is important, in contrast to other published pre-training approaches. Additionally, the image encoder, image resolution, and image token count have substantial impact, while the vision-language connector design is less critical.<\/p>\n\n\n\n

Based on these insights, the authors constructed the MM1 family of multimodal models of up to 30 billion parameters. This includes both dense models and mixture-of-experts variants. After supervised fine-tuning, the MM1 models demonstrate competitive performance on established multimodal benchmarks. The authors also note that the large-scale pre-training endows the MM1 models with enhanced in-context learning and multi-image reasoning capabilities.<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

Generative AI for designing and validating easily synthesizable and structurally novel antibiotics<\/a>, Stanford, McMaster University<\/em>.<\/p>\n\n\n\n

Presents SyntheMol, a generative model for drug design, optimized for creating easily synthesizable molecules. In contrast to models that evaluate molecules one-by-one, SyntheMol uses a combination of molecular property prediction models and Monte Carlo tree search to efficiently explore a vast combinatorial space of potential drug molecules. The Monte Carlo tree search algorithm allows SyntheMol to iteratively build up candidate structures by making sequential decisions about which molecular fragments to add or modify, guided by the model’s training on a large dataset of known drug-like molecules.<\/p>\n\n\n\n

\"\"\/<\/a><\/figure>\n\n\n\n

The researchers consciously built SyntheMol to design easy-to-synthesize molecules to enable wet lab validation, avoiding the common challenge of synthetically intractable AI-designed molecules that can\u2019t be tested in the real world. When tested, SyntheMol was able to generate six novel antibiotic compounds that displayed broad-spectrum antibacterial activity against multiple bacterial species, including A. baumannii, a bacterium with few known treatments. The code<\/a>, along with the data and models<\/a>, is fully open sourced. Jonathan Stokes, the main author, spoke at RAAIS in 2020 on the use of chemical biology and AI to develop novel antibacterial therapies. Check out the talk here<\/a> to see how this work\u2019s evolved. <\/p>\n\n\n\n

Also on our radar:<\/p>\n\n\n\n