Last week I've published zebra_simple on GitHub — simple LLMs reasoning tester, it measures LLMs' success rate of solving Zebra Puzzles based on ZebraLogic dataset.
We chose Zebra Puzzle as a good example of pure logic reasoning task for our research. But actually, I had one more reason — in childhood I loved to play Sherlock, so I wanted to recapture the feelings of childhood and please the models with this game 😉
Scheenshot from Sherlock 2.0 for MS DOS by Everett Kaser
Sherlock 2.0 game description from Mobygames:
Sherlock
The game screen is divided in several areas. The top-left area is the puzzle solve board. In the top-right area contain the horizontal or multi-column clues. The bottom row contain the vertical or 1-column clues.
A vertical clue tell the player that two squares are or are not in the same column. There are five types of horizontal clues. The "is next to" and "is not next to" clue says that two tiles are or are not adjacent to each other, but it doesn't say which one is on the left and on the right. The "is left of" clue (middle tile is yellow with 3 dots) says that one tile is to the left of another tile, but it doesn't say how far or close to the left.
The "is between" clue (three tiles with two sided arrow) says that three tiles are adjacent but this can be from left to right or from right to left. The "is not between" says the same except that the middle tile is not between the other two.
The zebra_simple compares N selected models success rate in solving same M Zebra Puzzle tasks and extracts Chain of Thoughts for the further analysis.
All models for all providers accessed via the same OpenAI Completions API, the most unified standard de-facto designed to query LLMs. For now we have tested models from Google, Groq, NVidia, OpenAI and DeepSeek but have also connectors to xAI, Anthropic and any custom providing compatible Completions API.
By default all models queried with temperature=0.6 and top_p=0.95 (except o3 and o1 not supporting these params) — this is the most recommended combination for logic reasoning, it could be changed in code now or added as command line arguments in future.
This tool is pretty simple to use and to modify for your own needs. Prompt template lives in zebra_template.py. This command runs 2 Zebra Puzzle tests of size 3x3 (#0 and #1 from the dataset) on llama-3.3-70b model by Groq with tokens limit 4k and writes results to the json log:
python -u zebra.py -m llama-3.3-70b-versatile@groq -s 3*3 -i 0,1 -t 4096
This is summary for 6 models comparison (number after the model name is success rate %):
{
"duration": "0:13:22.876821",
"size": "3*3",
"count": 5,
"max_tokens": 16000,
"models_summary": {
"gemini-2.5-flash-preview-05-20-none@google": 100,
"gemini-2.5-flash-preview-05-20-default@google": 100,
"nvidia/llama-3.1-nemotron-70b-instruct@nvidia": 40,
"nvidia/llama-3.1-nemotron-ultra-253b-v1@nvidia": 100,
"qwen/qwen3-32b-none@groq": 60,
"qwen/qwen3-32b-default@groq": 100
}
}
This is example of json log for one model, skipped parts listed as '...':
{
"model": "nvidia/llama-3.1-nemotron-ultra-253b-v1@nvidia",
"size": "3*3",
"max_tokens": 16000,
"total_tokens": 13804,
"started": "2025-06-17T15:40:48.655072",
"duration": "0:03:28.259337",
"count": 5,
"successed": 5,
"rate": 100,
"items": [
{
"id": "lgp-test-3x3-0",
"success": true,
"finished": "2025-06-17T15:42:05.356933",
"prompt": "\n\nTHE PUZZLE: ...",
"solution": { "House 1": ... },
"answer": { "House 1": ... },
"text": "",
"thought": "<think>\nOkay, let's tackle this logic puzzle step by step. First, I need to make sure I understand all the clues and how to translate them into formal logic. Let me start by breaking down the problem.\n\nWe have three houses, numbered 1 to 3 from left to right. Each house has ... the correct solution emerges.\n</think>"
},
"usage": {
"completion_tokens": 4037,
"prompt_tokens": 511,
"total_tokens": 4548,
"prompt_tokens_details": null
}
},
{
"id": "lgp-test-3x3-1",
...
},
...
]
}