استكشاف نماذج القرار من النوع الأول (System 1) في لعبة باكمان

معظم نماذج الذكاء الاصطناعي التي أعمل معها تُولّد نصوصًا. لكن Laya لا تفعل ذلك. Laya هي نموذج لاتخاذ القرارات من نوع "النظام 1" غير ذاتيّ الانحدار. فبدلًا من توليد إجابة، تمنحه حالة ومجموعة من الأسئلة محددة الأنواع، فيُرجع احتمالات للإجابات الممكنة.

أردت أن أرى كيف تبدو تجربة البناء باستخدامه فعليًا. لذا منحته لعبة باك مان.

كانت الفكرة بسيطة. في كل خطوة، يتحول الوضع الحالي لـ Pac-Man إلى تمثيل نصي صغير. تقرر Laya أي اتجاه تسلك، وتنفذ اللعبة الحركة. ثم نكرر العملية.

لم أكن أحاول بناء وكيل تنافسي للعبة باك مان. لم يتم تدريب Laya على لعب باك مان، ولم يكن الهدف من ذلك إجراء اختبار جاد لقدرات اللعب. كل ما أردته في الغالب هو معرفة ما الذي سيحدث عندما تضع نموذجًا لاتخاذ القرارات داخل حلقة مغلقة وتتركه يتخذ مئات القرارات بمفرده.

ما انتهيت إلى اكتشافه كان أكثر إثارة للاهتمام من مجرد معرفة ما إذا كان بإمكانه لعب اللعبة. كان للتغييرات الصغيرة في كيفية تمثيلي للحالة ووصف الحركات المتاحة تأثير كبير على نحو مفاجئ في سلوكه.

وبمجرد أن جعلت كل شيء يعمل، طبّقت الإعداد نفسه على نموذج آخر من نماذج النظام 1، وهو Jev. واتضح أن تلك التجربة الثانية كانت الجزء الأكثر فائدة. فقد أتاحت لي الفصل بين الأمور التي كانت بالفعل خصائص لهذا النوع من نماذج اتخاذ القرار، والأمور التي لم تكن سوى سمات خاصة بـ Laya.

التجربة

أنشأت نسخة صغيرة-مستقلة من لعبة باك مان، تضم متاهة بحجم 19×21، ونقاطًا وحبيبات طاقة، وأربعة أشباح بنسخ مبسطة من سلوكياتها المعتادة، والانتقال عبر الأنفاق، والأرواح، ونظام تسجيل النقاط. كما أن اللعبة مستقلة عن أي مكتبة للعرض، لذا تمكنت من تشغيلها بالكامل دون واجهة رسومية أثناء اختبار إعدادات مختلفة لاتخاذ-القرار. أُجريت جميع التجارب المحلية في Laya على بطاقة RTX 2060 بذاكرة VRAM سعتها 6 GB ومعالج Core i5-9400F.

في كل خطوة:

1. تنظر اللعبة إلى الحالة الحالية.

2. تُحوَّل تلك الحالة إلى تمثيل نصي موجز.

3. يتم تحويل الحركات المسموح بها إلى أوصاف للخيارات.

4. يختار النموذج اتجاهاً.

5. باك مان يتحرك.

6. تتكرر العملية.

إذًا الحلقة هي في الأساس:

حالة اللعبة ← تمثيل نصي ← قرار النموذج ← إجراء داخل اللعبة ← حالة اللعبة الجديدة

طرحت أيضًا على النموذج بعض الأسئلة الإضافية حول الحالة، مثل ما إذا كان Pac-Man في خطر ومدى خطورة الوضع الحالي. يستطيع Laya الإجابة عن عدة أسئلة محددة النوع في تمريرة أمامية واحدة، لذا تمكنت من الحصول على هذه المعلومات الإضافية دون الحاجة إلى إجراء استدعاءات منفصلة للنموذج.

الجزء المهم هو أن النموذج لا يرى اللعبة الفعلية مطلقًا. فهو لا يحصل على وحدات البكسل ولا على تمثيل منظّم للمتاهة. كل ما يعرفه عن الوضع الحالي يأتي من النص الذي أقدّمه له. واتضح أن ذلك كان مهمًا جدًا.

النتيجة 1: لدى الدولة ميزانية صارمة

كان حدسي الأول هو تزويد النموذج بأكبر قدر ممكن من المعلومات. تضمنت الحالة الأولية تمثيلًا للمتاهة باستخدام ASCII، ومعلومات عن المربعات المجاورة، ومواقع الأشباح والمسافات بينها وبين باك مان، واتجاه باك مان، والنتيجة، والأرواح المتبقية، والنقاط المتبقية، والحركات المسموح بها.

وكان شكلها التقريبي كما يلي:

{
  "view": ["#####", "#.. #", "#.P G", "# ..#", "#####"],
  "facing": "down",
  "neighbors": {
    "up": {"tile": "wall", "dots_ahead": 0},
    "left": {"tile": "dot", "dots_ahead": 3}
  },
  "ghosts": [
    {"at": [-2, 3], "dist": 5, "dir": "up", "frightened": false}
  ],
  "score": 340,
  "lives": 2,
  "dots_left": 120,
  "power_time_left": 0,
  "legal_moves": ["left", "right"]
}

بدا الأمر منطقيًا. فإذا كان النموذج يمتلك معلومات أكثر عن اللعبة، فستكون لديه معلومات أكثر لاتخاذ قراره. لكن كانت هناك مشكلة.

يبلغ الحد الأقصى لطول التسلسل في نقطة التحقق الإنجليزية الخاصة بـLaya's 512 رمزًا. وبعد احتساب التعليمات ونص الخيارات، لم يتبقَّ سوى نحو 330 إلى 380 رمزًا لحالة اللعبة. قست الحالة المذكورة أعلاه باستخدام أداة تقسيم الرموز الخاصة بنقطة التحقق نفسها. فكان عددها 341 رمزًا. وكان ذلك بالفعل قريبًا بشكل مقلق من الحد الأقصى.

ثم نظرت في كيفية تجميع المُدخل فعليًا:

[CLS] instructions [SEP] [MASK] option_0 [MASK] option_1 ... [SEP] state [SEP]

تُوضَع الحالة في نهاية التسلسل، وعندما يكون الإدخال طويلًا جدًا، يتم اقتطاعه من اليمين. وهذا يعني أن نهاية الحالة هي أول ما يختفي.

في إحدى الحالات التي كان فيها السياق مزدحمًا، فقد النموذج نهاية الحمولة:

..., "power_time_left": 0,
"legal_moves": ["up", "down", "left", "right"]

كان لا يزال بإمكان النموذج رؤية الحالة، لكنه لم يعد قادرًا على رؤية جميع المعلومات التي توقعت منه استخدامها. وقد أدى ذلك إلى وضع قاعدة مفيدة للتسلسل:

عندما يقتطع النموذج مدخلاته من اليمين، يكون ترتيب حقول الحالة لديك فعليًا قائمة أولويات.

يجب أن تظهر المعلومات الأكثر أهمية أولاً. أما المعلومات التي يمكن الاستغناء عنها بأمان، فينبغي أن تأتي لاحقاً. لذا، بدلاً من التعامل مع الحالة باعتبارها قاموساً عادياً، بدأت في بنائها من الأكثر أهمية إلى الأقل أهمية. وبهذه الطريقة، تصبح الحقول التي يعتمد عليها القرار فعلياً غير قابلة للاقتطاع في البداية.

لكن حل مشكلة حدّ الرموز كشف عن مشكلة أخرى. فقد كنت أزوّد النموذج بكمية كبيرة من المعلومات المكررة.

النتيجة 2: أدت المزيد من المعلومات إلى تفاقم الحالة

Looking at the original state, I realized that I was describing the same situation several times. The ASCII map described the board. neighbors described what was around Pac-Man. ghosts described the threats. Then the option text described the neighboring tiles again.

I wasn't really giving the model more information. I was giving it the same information in several different forms. So I tried the opposite approach. I gave it only the information I thought actually mattered for the next decision.

The new state looked more like this:

{
  "legal_moves": ["up", "down", "left", "right"],
  "situation": "No power pellet is active, so touching a ghost costs a life.",
  "threats": [
    "a dangerous ghost 3 tiles away, 2 up and 1 left, heading down toward Pac-Man"
  ],
  "facing": "down",
  "dots_left": 177
}

The difference was much bigger than I expected. The original representation was 341 tokens. The compact version averaged only 66 tokens, and even in situations with several nearby ghosts it stayed well below the limit. Across 60 sampled states, its worst case was 106 tokens.

Truncation stopped being something I had to think about. The compact state was also easier to reason about. Instead of asking the model to reconstruct what a collection of coordinates and counters meant, I could tell it directly what was relevant to the decision.

For example, instead of:

"power_time_left": 0

I could say:

"No power pellet is active, so touching a ghost costs a life."

The second representation doesn't contain more information. It's just closer to the actual decision being made. That was my first real lesson from the experiment:

A better state is the smallest representation that still contains what the decision depends on.

Finding 3: The answer depends on the order of the options

Once the state was under control, I started looking at the actual decision. I had assumed that if the state stayed the same, changing the order in which the options were presented shouldn't matter. So I tested it. I took one four-way junction, kept the state and option text completely fixed, and asked the same question under all 24 possible orderings of the four options.

The semantic question was identical every time. Only the order changed. The resulting decisions were:

up 2    right 10    down 6    left 6

The model's confidence was extremely low, between 0.001 and 0.04, so the probability distribution was almost flat. That meant the final choice was sometimes being decided by presentation rather than by a meaningful difference between the options. Slot 0 was selected 9 times out of 24, even though each option should have appeared there exactly 6 times.

This was especially interesting because Laya can answer multiple questions in a single forward pass. So instead of running all 24 permutations separately, I could rotate the options:

rotations = [
    legal[i:] + legal[:i]
    for i in range(len(legal))
]

and ask one question for each rotation in the same call.

Each option appears in every position exactly once. Then I average the probabilities across rotations. This removes the positional effect without requiring 24 separate model calls.

I also started tracking something I called order spread: how much an option's probability changes across the different rotations. A low order spread means the model is making a decision based mostly on the content. A high order spread means presentation is having a large influence on the answer. This ended up being one of the most useful diagnostics in the experiment.

The problem with determinism

Once I had the smaller state representation working, Pac-Man started doing something I hadn't expected. It got stuck. Not because the model crashed. Not because it was confused. It was doing exactly what I had asked it to do.

Laya is deterministic. Give it the same state and the same questions, and it gives you the same answer. Most of the time, that's a useful property. It makes the system reproducible and means that running the same experiment twice doesn't introduce randomness into the decision process. But in a game, determinism has an awkward side effect.

Pac-Man would sometimes enter a small corridor and start going back and forth between the same few cells. Once it reached the same state again, Laya made the same decision. That decision led it back into the same state. The cycle repeated forever. The agent had effectively become a finite state machine that had entered a cycle. There was no randomness that could eventually knock it out of the loop.

Interestingly, the original, much larger state had hidden this problem. It contained the exact coordinates of the ghosts, and because the ghosts were moving, the state was slightly different on every visit. That accidental variation was enough to occasionally break the cycle. Once I cleaned up the state, the problem became obvious.

So I needed the environment itself to detect when Pac-Man was stuck. I added a loop detector based on recent movement history. I tested several thresholds:

The final rule was:

If the last 16 positions cover 4 or fewer distinct cells, treat it as a loop.

It never fired during ordinary random movement in my test and still caught a substantial fraction of the deliberately stuck behavior. I also had to make the escape behavior sticky. The first version would escape the corridor and then immediately walk back into it one step later. So after breaking a loop, the move that would immediately reverse the escape was temporarily blocked.

This wasn't really about making Laya smarter. The model was doing exactly the same thing it had done before. I was adding a small amount of memory to the environment around it. And that turned out to be an important distinction for the rest of the experiment.

A deterministic decision model doesn't remember that it has already made a bad decision unless that information is represented in its state.

So I stopped treating the loop as a model failure and started treating it as part of the system I was building. The model decides what to do. The environment keeps track of what has already happened.

Finding 4: The wording of an option changes the decision

This was probably the strangest behavior I found. At one junction, one direction had four dots ahead while another was an empty dead end. I expected the model to strongly prefer the direction containing the dots. Instead, with the original option descriptions, it often preferred the dead end.

The options looked roughly like:

down:  "next tile empty, 0 dots this way, is a turn"
right: "next tile dot, 4 dots this way, is a turn"

The model chose the dot-rich option only about 30% of the time in the original setup. So I started changing the wording while keeping the underlying information the same. I tested several versions:

The experiment was averaged over six junctions and twelve ordered label pairs so that neither the direction label nor its position could explain the result. 0.50 is chance. The difference was substantial.

The model seemed much more responsive when the option description contained an explicit judgement. Compare:

next tile dot, 4 dots this way

with:

a good move: eats a dot now and more dots lie ahead this way

The underlying information is almost identical. The second one simply tells the model what that information means for the decision.

So I changed the option builder to produce descriptions such as:

"a deadly move: a dangerous ghost is waiting this way"

"an excellent move: a power pellet can be eaten this way"

"a good move: eats a dot now and more dots lie ahead this way"

"a poor move: an empty tile with nothing to eat ahead this way"

For Laya, that made a large difference. But this is where the second model became important.

Is this actually a property of decision models?

At this point, I had learned a lot about how to make Laya behave. But that isn't necessarily the same thing as learning how to work with decision models. Maybe the sensitivity to option wording was a property of System 1 models. Or maybe it was just a strange property of this particular checkpoint. So I wanted a control.

I took the same Pac-Man environment, the same state representation, the same option descriptions, the same loop detector, and the same evaluation code. Then I swapped out Laya for Jev. The important part was that I didn't build a second Pac-Man agent. I put both models behind the same interface:

class Backend:
    def predict(self, state, questions):
        ...

Everything above that interface stayed the same. That gave me a much cleaner comparison. If the two models behaved differently, the model itself was the thing that had changed.

What happened in actual Pac-Man?

Once the decision-level experiments were done, I let the models play. I gave each agent five episodes, using the same five seeds and a maximum budget of 600 decisions per episode. I also included a random walker as a baseline.

Episode results

The score is useful, but it isn't the whole story. distinct cells tells me how much of the maze the agent actually explored. loop rescues / 100 decisions tells me how often the environment had to intervene because the model got stuck. Those metrics make some of the behavioral differences much easier to see.

Jev explored 91.6% of the maze compared with Laya's 56.2%. It also needed the loop detector much less frequently. The random walker never triggered the loop detector, which isn't necessarily a good thing. It simply doesn't tend to spend sixteen consecutive moves circling the same few cells. And every agent eventually lost all three lives.

So this isn't a "who finished Pac-Man?" benchmark. It measures how the different decision policies behave before the ghosts eventually win.

Decision-level results

Episode scores can be noisy, so I also compared the models directly on smaller controlled probes. I used 48 ghost-avoidance scenarios and 12 states sampled from random walks, with byte-identical inputs for both models.

The ghost-avoidance result was particularly striking. Jev did not walk into the ghost in any of the 48 scenarios. Laya chose the fatal direction 37.5% of the time, compared with a 46.2% chance baseline.

But the most interesting result for me was the wording experiment. Remember the change that made a big difference for Laya? The verdict-style option descriptions. Jev didn't need it. Jev scored 0.735 with the simpler terrain description, compared with 0.718 with the verdict-style description. So the thing I initially thought might be a general lesson about how System 1 models understand options turned out to be a Laya-specific workaround. That distinction is exactly why I wanted the second model.

The speed tradeoff

The behavioral difference came with a fairly obvious engineering tradeoff. Laya averaged around 71 ms per decision in the decision-level benchmark. Jev averaged around 361 ms. So Jev took a little over five times longer per decision.

The comparison is also not completely apples-to-apples from a deployment perspective. Laya runs locally on the RTX 2060. Jev is accessed over the network. So Laya's latency is a local forward pass while Jev's includes a network round trip. That matters if you're building an actual real-time agent. The point isn't that one number is universally better. It's that the same Pac-Man environment exposed a very different set of tradeoffs between the two models.

The cost of the experiment

The Jev API charged for input tokens. The decision pipeline used roughly 630 input tokens per decision after accounting for the option rotations and auxiliary questions. A full episode cost around $0.013. The five-episode comparison cost roughly $0.064, and the decision-level probe cost around $0.004. The entire comparison in this post came to roughly seven cents. That made it cheap enough to experiment with things like option ordering and alternative prompts rather than treating every model call as something precious.

What actually generalized?

After running the same probes against both models, I ended up with two very different categories of findings. Some appeared in both models. Others were clearly specific to Laya.

Findings that appeared across both models

State representation matters. A huge state isn't automatically a better state. Removing redundant representations reduced the state from 341 tokens to roughly 66 without making the game state less useful.

Serialization matters. When a model right-truncates its input, the order of your fields becomes a priority list.

Option order matters. Both models changed their decisions when the same options were presented in different orders. Rotating the options within a single batched call provides a cheap way to reduce that positional effect.

Option labels can have their own priors. Both models showed a strong preference for certain direction labels even when the option descriptions were identical.

Determinism needs to be designed around. A deterministic model can enter a loop and stay there forever if the state it sees becomes identical. That is something the surrounding agent architecture needs to handle.

Decision-level probes are more informative than long episodes. A Pac-Man score contains the accumulated effects of hundreds of decisions. A carefully designed single-state probe can isolate one behavior and tell you much more about why the model is doing something.

Findings that turned out to be Laya-specific

Verdict-first option descriptions. Laya benefited substantially from descriptions such as:

"a good move: eats a dot now and more dots lie ahead this way"

Jev didn't. In fact, Jev performed slightly better with the simpler terrain description. So I wouldn't generalize this into a rule about all decision models.

Explicitly stating consequences. I also found that saying:

"touching a ghost costs a life"

seemed more useful than:

"power_time_left": 0

What I actually learned

The interesting part was how much of the agent's behavior was determined by the interface between the environment and the model. The model never saw Pac-Man. It saw a description of Pac-Man. And small decisions about that description had large effects:

• Which information gets included?

• In what order?

• How much redundant information is there?

• What do the option names look like?

• In what order are those options presented?

• Does the option description merely describe the world, or does it explain what that information means for the decision?

• What happens when the model sees exactly the same state again?

Those are not really Pac-Man questions. They're questions about how to turn an environment into something a decision model can reason about. And the Jev comparison made one thing clear: you have to be careful about turning a model-specific quirk into a general rule.

The biggest improvement I found with Laya, rewriting the option descriptions to include explicit judgements, almost disappeared when I switched models. Meanwhile, the option-order effect and direction-label bias showed up in both. That distinction would have been impossible to see if I had stopped after getting Laya to work.

Getting a decision model to make a choice is easy. Getting it to make useful decisions consistently is much more about the system you build around it.