ChatGPT: Smarter Than a Fifth Grader (at Fractions)

Curiosity got the better of me when I followed up on ChatGPT's improved math capabilities as an application to Hewitt's fractional exercise. When I ran the Hewitt fractional task against the newest ChatGPT model (o1), it successfully provided the process and solution in short order. I then asked the model to offer alternative methods to arrive at the solution, and it managed to do so effectively. The results are posted at the bottom of this post. The short answer to why the platform has improved is likely due to recent advancements in training data and enhanced reasoning capabilities that have upgraded the model's accuracy with mathematical queries. Feel free to stop reading here; the following might be more boring for you than it was for me.

Context: Why ChatGPT Struggled Previously

When I reflected on ChatGPT's past failure to solve a seemingly simple question like Dave Hewitt’s fractional exercise, it seemed like an interesting benchmark for assessing how ChatGPT's mathematical capabilities have evolved and why they previously fell short. Below is a brief analysis of what might have been lacking when the model failed to find the right solution last year:

• Reasoning: ChatGPT's reliance on text-based training limited its capacity for precise step-by-step reasoning. Human learners can adapt strategies, discover new approaches, and iteratively apply patterns, a capability that previous versions of ChatGPT struggled to emulate.
• Training Data on Rational Numbers: The model likely needed more comprehensive mathematical training data, which is now incorporated into recent versions.

As Susan pointed out, Wolfram Alpha has solved similar problems for many years. This comparison highlights the distinction between LLMs and platforms like Wolfram Alpha. Wolfram Alpha leverages curated mathematical databases and processing algorithms powered by the Mathematica engine, excelling at precise solutions without extensive context. This approach contrasts sharply with LLMs like ChatGPT. (As an aside, Wolfram Alpha now has its own ChatGPT implementation)

Key Improvements

Over the past six months, several enhancements have been made to ChatGPT 3.5 and 4.0 to improve their ability to handle mathematical data:

• Enhanced Training Datasets: The models have been retrained with more comprehensive datasets, encompassing a wider variety of mathematical problems and solutions. This broader exposure has improved the models' grasp of mathematical concepts, operations, and problem-solving techniques.
• Algorithmic Enhancements: Adjustments to the underlying algorithms have enhanced numerical computation capabilities.
• Incorporation of Mathematical Reasoning Frameworks: The models now utilize frameworks that prioritize logical reasoning patterns, leading to more robust problem-solving.
• Feedback-Driven Refinements: User feedback identifying specific errors or limitations has been systematically analyzed to fine-tune the models, improving overall mathematical performance.

These targeted changes have collectively enhanced the accuracy of ChatGPT 3.5 and 4.0 in processing mathematical data, resulting in better logical reasoning and more consistent performance in tasks like solving fraction problems.

New Model: ChatGPT o1

ChatGPT introduced a new model in September known as ChatGPT o1, representing a paradigm shift. This model incorporates chain of thought (CoT) reasoning, allowing it to break down complex problems into manageable steps. It is designed to emulate human-like reasoning and can offer detailed, step-by-step explanations for complex problems. Earlier models required specific prompts to engage in step-by-step reasoning. However, o1 is designed to do this more naturally, offering detailed explanations for intricate problems and excelling in fields such as mathematics, science, and coding.

OpenAI's o1 model also uses process supervision, a training method that provides feedback on each solution step rather than only the final answer. This leads to better performance in complex problem-solving. To support this, OpenAI compiled the PRM800K dataset, featuring 800,000 labeled steps across 75,000 solutions.

These innovations mark a shift from models merely retrieving memorized answers to ones that can recall and apply learned reasoning steps, enhancing problem-solving capabilities and unlocking new potential.

Takeaway

Should we trust these LLMs? I recently listened to a podcast featuring AI professional Baris Gultekin from Snowflake, a cloud-based data analysis platform. When asked about the current state of these systems' accuracy, he offered an insightful perspective:

“I try to see where we are, this generative ability, the creativity, if you will, as a feature to a certain degree. The type of use case that is great is when you ask the language models to generate something, to generate content. And if my question is a factual question, then I know to be careful.”...“If an LLM is provided with grounding, if an LLM is provided with the data, it does not hallucinate. Only when there is a lack of information.”

This aligns pretty well with my thinking on generative AI. Leaning into the generative and creative aspects of these tools is amazing. Still, it's always necessary to be able to verify the quality of what these tools generate when seeking facts.

If you're still reading, I'm sorry it took so long. Honestly, I likely wouldn’t have attempted this length a summary without ChatGPT’s assistance in organizing and contextualizing my thoughts. I hope this post helps clarify our conversation about generative AI and its applications in math. At the very least, we now know generative AI can correctly find that 11/15 is the only fraction between 5/7 and 3/4 with a numerator of 11. So proud :)