Earlier this year, Meta made a significant open-source contribution with Llama2, an alternative to ChatGPT. They released 3 models, one with 7 billion (7B) parameters, on with 13 billion parameters (13b) and one with 70 billion parameters (70b). For reference, the 70B model performs similar to / slightly worst than the Chat-GPT 3.5 model depending on which benchmark you read. We've deployed the 2 smaller models - Llama2 7B and 13B - and had a small benchmark.
We used 2 queries to test the chat-variation of the Llama2 13b and 7b models:
The outputs from the models are at the bottom of this post. Note that the pirate-speak is only half-a-joke, we intentionally wanted to test how the LLM responds in a style that doesn't match the vast amount of it's training data.
Ultimately it boils down to an age-old principle: using the right tool for the right job. Certain areas of your use-case may require a simple interaction and a 7B LLM is the best choice: offering speed and simplicity and low cost (for example summarizing large emails to a few bullet points).
But there may be some areas of your use-case that require more sophistication (for example drafting an email response to certain types of emails), and then a 13B or 70B model need to do the heavy-lifting
We'll continue to learn how to run, fine-tune and train these models. It's incredibly exciting that this technology is available to build solutions on.
Approach
We used 2 queries to test the chat-variation of the Llama2 13b and 7b models:
- "I need instructions for a Christmas soup. Write as a pirate and use funny metaphors"
- We requested a summary from the first 11 paragraphs of this nytimes article: approx. 600 words
The outputs from the models are at the bottom of this post. Note that the pirate-speak is only half-a-joke, we intentionally wanted to test how the LLM responds in a style that doesn't match the vast amount of it's training data.
Findings
- First of all, we're incredibly proud that we can run these models on our local infrastructure at good performance. This opens the door to new state-of-the-art solutions for our customers while guaranteeing that their data remains in Switzerland and adheres to the highest security standards. We'll be deploying the 70B model shortly and building up a cluster of Llama2 LLMs
- 13B is better, the summary is more incisive and the pirate recipe flows better (and we actually want to cook it). However ...
- 7B didn't do bad, and 7B can run on only 1 24 VRAM GPU whereas 13B needs to be distributed on two. I.e. 7B could process twice the amount of queries that 13B does considering similar hardware (more-or-less, we're simplifying)
Conclusion
Ultimately it boils down to an age-old principle: using the right tool for the right job. Certain areas of your use-case may require a simple interaction and a 7B LLM is the best choice: offering speed and simplicity and low cost (for example summarizing large emails to a few bullet points).
But there may be some areas of your use-case that require more sophistication (for example drafting an email response to certain types of emails), and then a 13B or 70B model need to do the heavy-lifting
We'll continue to learn how to run, fine-tune and train these models. It's incredibly exciting that this technology is available to build solutions on.
Output 13b - pirate soup query
- Ahoy matey! Yer lookin' fer a Christmas soup, eh? Well, hoist the sails and set course fer a hearty, swashbucklin' soup that'll make ye feel like the scurviest of scallywags!
- First, ye'll need to gather yer ingredients like a treasure chest filled with gold doubloons. Ye'll need some onions, garlic, carrots, potatoes, and a wee bit o' beef or seafood (arrr, make it count, matey!). Now, don't be a landlubber, chop those veggies up like ye're singin' a sea shanty!
- Next, ye'll need to cook up yer soup like ye're sailin' the high seas. In a great big pot, mind ye! Add in some olive oil like a drop o' rum, and let it swirl like the ocean currents. Throw in yer chopped veggies and cook 'em until they're soft as a sea breeze.
- Now, here be the fun part, me hearty! Add in yer beef or seafood and cook it until it's browned like a well-worn leather sea chest. Don't ferget the spices, me matey! A pinch o' salt, a grind o' pepper, and a dash o' paprika'll make yer soup taste like the treasure of the seven seas!
- Now, let it simmer like a ship at anchor for a good long while, until the veggies be soft as a mermaid's song and the meat be fallin' off the bone like a piece o' booty.
- And that's it, me hearty! Yer Christmas soup be ready to be served like a treasure chest overflowin' with booty! Serve it up with some crusty bread like a ship's biscuit, and a mug o' grog like a pint o' rum.
- So hoist the colors, me hearties, and enjoy yer Christmas soup like the scurviest of scallywags! Arrrr!
Output 13b - nytimes article summary
- Sam Altman, CEO of OpenAI, was unexpectedly fired by the company's board members via a video call on November 17th.
- The ouster was the culmination of longstanding tensions within OpenAI, pitting those concerned about AI's power against those seeking profit and prestige.
- The drama has raised questions about the trustworthiness of OpenAI and the AI industry as a whole, as well as the motivations and ethics of its leaders and investors.
Output 7b - pirate soup query
- Ahoy matey! Here be the instructions for a swashbucklin' Christmas soup, fit fer a scurvy dog like yerself!
- Step 1: Ye'll be needin' some o' that fancy-schmancy "vegetable broth" (translation: water with a bit o' salt and some veggies floating in it). Don't be forgettin' the "cream o' mushroom soup" (translation: a can o' mushroom soup, because ye don't want to be eatin' nothin' too fancy on a first date). And o' course, ye can't forget the "chopped ham" (translation: a bit o' ham that's been chopped up into bite-sized pieces).
- Step 2: Chop up yer ingredients like a pirate choppin' up a treasure chest. Take yer veggies and chop 'em up into little pieces that'll fit in yer mouth. Don't be worryin' about how big or small they are, just make sure they're all roughly the same size so ye can eat 'em easy. And when ye get to the ham, make sure ye chop it up into bite-sized pieces so ye can eat it easy too.
- Step 3: Cook yer soup like a pirate cookin' up a feast. Pour yer broth and mushroom soup into a big ol' pot and heat it up on the stove. Don't be afraid to add a bit o' seasonin' (translation: salt and pepper) to give it a bit o' flavor. And when ye add the chopped ham, make sure ye stir it in good so it's nice and hot all the way through.
- Step 4: Serve yer soup like a pirate servin' up a treasure chest full o' booty. Pour yerself a big ol' bowl and enjoy! Don't be forgettin' to add a bit o' crusty bread or crackers on top to soak up all the goodness. And if ye be feelin' extra hungry, ye can always add a bit o' shredded cheese or a dollop o' sour cream on top to make it extra special.
- So there ye have it, me hearty! A swashbucklin' Christmas soup that'll make ye feel like a pirate on the high seas. Enjoy yer feast, and don't be forgettin' to save me a bit o' the loot!
Output 7b - nytimes article summary
- Sam Altman, the CEO of OpenAI, was fired from the company after a 15-minute video call with the independent board members who had been secretly plotting against him for months.
- The board members accused Altman of dishonesty and voted to remove him from OpenAI, which he co-founded.
- Altman's ouster was the culmination of years of tensions at OpenAI, where divisions deepened between those who were concerned about AI's power and those who saw it as a profit and prestige opportunity.