Steam Review Sentiment Analysis: Fine-Tuning BERT on 229,102 Game Reviews
NLP · BERT · Fine-tuning · KerasNLP · TensorFlow · FastAPI · Docker
Where the idea came from
Press review scores have become hard to read. What really separates a 7/10 from an 8/10? Some games are also rated very highly by critics but hated by players, and the latest 3D Pokémon games are the clearest example. When a new game often costs €70 at launch, a wrong call is expensive.
So I wanted another way to judge a game: take into account what players think overall, how they actually felt playing it, rather than a score that is bound to be subjective.
Those reviews already exist. On Steam, a popular game has tens of thousands of them, far too many to read, and the share of positive ones doesn't tell you what people like or what annoys them. The idea is a tool that could read those reviews for me, then follow how players feel over time: after a trailer, during a beta, after a big update. And not only on Steam, but also on other sites or under a YouTube video.
The first building block is a sentiment analysis model: it reads a review and says whether it is positive or negative. It has to be specialised and accurate, so it can spot reliable trends at low cost. That's what this article is about.
Why not use an off-the-shelf model?
Most available sentiment models are trained on movie reviews, tweets or product reviews. Gamers have their own way of writing, with a lot of irony. Here are three real reviews from the dataset:
- "Roadmap Simulator 2019 Snail paced development with constant delays and promises": negative, about a game whose development drags on.
- "best played with a membrane keyboard, a rollerball mouse, and a 800x600 CRT": positive, a nod to how old the first Counter-Strike is.
- "I like this game a lot, however I wouldn't really recommend it to anyone either": negative, despite the "I like this game a lot".
A tool that counts positive and negative words gets this kind of sentence wrong easily. So I took an existing language model, BERT, and specialised it on gamer reviews. That's called fine-tuning.
A simple question
As in my PS5 discount project, the question has a yes or no answer (a binary classification): is the review positive?
The good news is that the answer is already there. On Steam, every review comes with a thumbs up or down: the player recommends the game, or not. Nothing to label by hand, this is supervised learning with labels provided by the players themselves.
Starting from 100 million reviews
The starting point is the Steam Reviews dataset on Kaggle: about 100 million reviews in every language, 40 GB of CSV. Far too much to load into memory. So I read the file in chunks, with good old Python and pandas, and extracted one file per language. The dataset author also provides a lighter file of about 500,000 reviews, which I used for exploration.
Before using a dataset found online, check its licence and terms of use, especially for commercial use.
What the exploration shows
Exploratory data analysis is about checking that the data makes sense and spotting traps before training anything. Across the 193,174 English reviews of the lighter file:
- 81% of reviews are positive. Trained on this data as is, a model could get away with answering "positive" almost every time and post a good score without understanding anything. So the two classes will need rebalancing to 50/50, as many positive reviews as negative ones.
- A review is 190 words long on average, but 22% have fewer than 10 words and 12% have more than 512. Under 10 words, the signal is weak: you rarely get a structured, reasoned opinion.
- 4,421 reviews contain almost no letters. They are ASCII art drawings, like a cat drawn with characters.
- The vote score (weighted_vote_score) is computed by Valve, which doesn't publish the method. It mostly follows the number of "helpful" votes (Spearman correlation of 0.62). So it measures how credible a review is, not its sentiment.
I also read random reviews from each class to check that the labels made sense. They do, with a lot of irony. All the figures are in the project's exploration notebook.
Cleaning and filtering
The exploration gave me the cleaning rules:
- For English, I first kept reviews with a vote score above 0.6. At 0.5, the file was still over 6 GB. For French, which is much rarer, I took every review and then applied a 0.5 threshold.
- To limit bots and biased reviews, I only kept reviews from legitimate players: they actually paid for the game on Steam (verified purchase, not received for free) and played it for more than 30 minutes.
- I removed HTML and BBCode tags (such as
[h1]or[b]), then the reviews where more than half of the characters aren't letters. - I kept reviews of 10 to 400 words. Beyond 400 words, the model would cut the text anyway during fine-tuning.
- Finally, I balanced the classes to 50/50 by randomly dropping positive reviews (undersampling).

My initial goal was at least 50,000 reviews per language. I ended up with 229,102 in English and 60,420 in French, plenty to work with.
Now on to building the specialised model, through fine-tuning. The idea: start from a language model that is already trained, then retrain it on these reviews so it learns how gamers write. For this kind of task, one of the recommended models is BERT.
Why BERT
Heads up: what follows is more technical and assumes some deep learning basics. If you want to learn, the Alyra school offers courses on the topic (in French).
BERT is a language model released by Google in 2018 and pre-trained on Wikipedia and books. Thanks to the Transformer architecture, it reads each word in light of what comes before and after it. It can be reused and specialised for a specific task: that's transfer learning.
I used the ready-made models (presets) from KerasNLP, since renamed KerasHub, with TensorFlow. I tested four sizes:
| Model | Parameters |
|---|---|
| BERT Tiny | ~4 million |
| BERT Small | ~29 million |
| DistilBERT Base | ~66 million |
| BERT Base | ~110 million |
DistilBERT is a compressed version of BERT Base, 40% smaller, that keeps most of its performance. All training ran on a single Nvidia RTX 4070 SUPER graphics card, with 12 GB of memory. That's a minimum for this kind of experiment, and 16 GB or more makes life easier.
The review is split into tokens and encoded by BERT, then a small classification head returns a probability between 0 (negative) and 1 (positive). The code is in the fine-tuning notebook.
![Architecture: Steam review, KerasNLP preprocessor (WordPiece tokens, 128 or 256 tokens), frozen or retrained BERT encoder, [CLS] token, Dropout and Dense classification head, 73% positive probability](/blog/steam-reviews-bert/model-en.webp)
Freeze BERT or retrain it?
First test, on a sample of 20,000 reviews: 80% for training, 10% for validation and 10% for testing. I compared two approaches.
- Feature extraction. BERT Small is frozen and only the classification head learns, barely 33,000 parameters out of almost 29 million. Result: 79.0% accuracy.
- Fine-tuning. The whole model learns again, with a very small learning rate (2e-5) so it doesn't erase what BERT already knows. Two callbacks, EarlyStopping and ReduceLROnPlateau, stop training before overfitting. With BERT Tiny, even though it's six times smaller: 81.9%.
So a small retrained model beats a bigger frozen one. BERT's generic representations aren't enough, it has to adapt to how gamers write. I kept fine-tuning for everything that followed.
Scaling up, phase by phase
On the full dataset, training a big model takes more than an hour. So don't waste compute time on your very first fine-tuning run. Start with a small dataset and a small model, test your initial settings, then scale up step by step, keeping only the best hyperparameters. That's what I did, in several phases. Every run is logged in a CSV file (settings, score, duration), which makes comparisons easy.
Phase 1, finding the right settings. On 50,000 reviews with BERT Small, I tested 27 hyperparameter combinations: 3 learning rates, 3 classification heads and 3 callback strategies. Each run takes 2 to 5 minutes. Scores range from 87.2% to 88.3%: here, the settings barely matter.
Phase 2, the full dataset. I ran 6 of these combinations again on all 229,102 reviews, still with BERT Small, in 12 to 15 minutes each. Scores rise to between 89.8% and 90.2%. With four times more data, you gain 2 points.
Phase 3, bigger models. With the best settings, BERT Base reaches 91.9% after 1 h 17 min of training. DistilBERT does almost as well, 91.2%, in half the time.
Phase 3.1, review length. At 128 tokens, long reviews get cut. So I trained DistilBERT with sequences of 128, 256 and 512 tokens.

Moving to 256 tokens gains almost 2 points. Moving to 512 doubles the training time for no gain at all. In the end, DistilBERT at 256 tokens beats BERT Base with 40% fewer parameters (BERT Base was only tested at 128 tokens).

All the results, phase by phase, are in the training strategy notebook.
Is 92.9% good?
Both classes are balanced, so random guessing would give 50% accuracy. That's our point of comparison (the baseline).
Out of the 22,911 reviews in the test set, the model gets 1,624 wrong. Some reviews in the dataset are ambiguous even for a human, like "I flipped a coin to decide whether to give this a positive or negative review", marked negative by its author.
With fine-tuning, you also need to know that past a certain level, gaining one more point takes disproportionate resources.
And against a generic tool? VADER is a classic tool based on a dictionary of positive and negative words. On the review "It's decent, nothing groundbreaking.":
- VADER returns a score of exactly 0, neither positive nor negative.
- Fine-tuned BERT Small hesitates: negative, at 53%.
- Fine-tuned BERT Base gets that "decent" leans positive: positive, at 73%.
To take the study a bit further, we could imagine evaluating a generic DistilBERT, trained on movie reviews, on the same test set. That would put a precise number on the gain from fine-tuning.
Where to stop?
Bigger models, like RoBERTa or DeBERTa v3, often get better scores. But every extra point has a price: longer training, slower answers and more expensive hosting. The 512-token test shows it well, with twice the compute for zero extra points. Don't forget about production either, and the use case the business actually asked for. Here, the end goal is to analyse thousands of reviews: the specialised model also has to save on cost and inference speed. A light model that is almost as accurate is often the best choice.
An API to use it
To use the model from another application, here is a quick and simple example: I exposed it through a REST API built with FastAPI, in a Docker container. A POST /predict request with the review text:
{
"text": "It's decent, nothing groundbreaking."
}
returns the sentiment and the model's confidence:
{
"sentiment": "POSITIVE",
"confidence": 73.24,
"probabilities": {
"negative": 26.76,
"positive": 73.24
}
}
To put the API online, you can use services like AWS SageMaker, whose serverless mode is billed per use, or host the Docker container on AWS ECS.
What about French reviews?
The French dataset of 60,420 reviews is ready. The models used so far only know English, so a different base is needed. The pipeline is already set up for a multilingual BERT (bert_base_multi), and I'll then compare it with CamemBERT, a model trained on French. For these heavier models, an RTX 4070 SUPER and its 12 GB of memory are no longer enough locally: a more powerful card or the cloud will be needed.
If you want to try it yourself
A few tips from this project:
- Start small. 20,000 reviews and BERT Tiny are enough to get a first result quickly and validate the approach.
- Read your data by hand. That's how I spotted the ASCII art, the irony and the reviews that were too short.
- Check what your label actually measures. Here, the vote says "I recommend it", which doesn't always match the tone of the text.
- If your field has its own vocabulary, retrain the model instead of freezing it.
- Log every run (settings, score, duration) and watch the cost. A setting that doubles compute time without adding anything isn't worth it.
What's next
The next step is the French model, then a measured comparison with a generic model on the same test set. After that come deploying the API to the cloud and collecting reviews from other sources (gaming sites, YouTube comments). And why not an app to follow how players feel about a game, day after day.
The whole project is on GitHub: the exploration, dataset creation, fine-tuning and deployment notebooks, the training scripts, the results of each phase and the API code.
Thanks to the Alyra team for the resources and advice provided during this project, and to the author of the Steam Reviews dataset on Kaggle.
Steam is a registered trademark of Valve Corporation and TensorFlow is a trademark of Google LLC. This is a personal project, not affiliated with or endorsed by Valve.
The code and results published on GitHub are provided for educational purposes. They may contain errors and shouldn't be used as they are. For any use other than personal or educational, please get in touch first.