Ray Uzwyshyn Ph.D. MBA MLIS
Testing the 'Real' Coding Limits of Claude 3.7 Sonnet through Basic Data Visualization Programming Benchmarking Tests
In the middle of March 2025, I embarked on what I thought would be a straightforward benchmarking exercise: creating an interactive financial data dashboard using Anthropic's latest large language model, Claude 3.7 Sonnet. From what I had heard, Claude was 'the best' in the world currently at AI coding. We were one step away from artificial general intelligence and two steps from Artificial SuperIntelligence. This would eliminate almost all human coders in the world, if Dario Amodei, CEO of Anthropic was to be believed. These were also some of the latest more hyperbolic pronouncements on podcasts and rebroadcast by AI Youtube video influencers. What unfolded though in my AI programming benchmarking adventures over nearly two hours revealed both the remarkable analytical capabilities of today's best AI models but also very significant technical limitations of today's most advanced AI systems. This revealed dichotomy overviewed below can tells us much about where we stand in the Spring of 2025 on the path to artificial general intelligence in the 'real' world.
The complex but worthy financial modeling and pragmatic 'global maket' real world problem task seemed well-defined: visualize the relationship between stock market technical analysis recessionary "death crosses" in major market indexes (when short-term moving averages cross below long-term ones), and US Federal Reserve interest rate decisions, and subsequent market historical performance from the dotcom bust of 2000 to the present day. A difficult enough problem for a Masters programming student or junior market analyst but not rocket science. This analysis would be a good test though require synthesizing complex financial concepts, identifying patterns across multiple market cycles, and creating basic dat visualizations that would make these relationships clear for better insights into current market gyrations and general trends with the all mighty US federal reserves and fed funds rate.
The Original Prompt
"Could you create some charts and tables with regards to previous recent historical periods and the death cross and major indexes and the Fed beginning to lower interest rates in relationship with these falling indexes?" I prompted Claude. "I wish to look at these relationships. Use dates for this and also create this as interactive dashboards to see these relationships with the major indexes and associated ETFs: SPY, QQQ, IWM, DJI, NYSE. Also, use data from the present historical period. Go back to the dotcom bust and forward from there. Pay attention to color and information visualization principles so this shows this relationship."
Claude began enthusiastically, immediately searching for relevant historical data on death crosses, market performance, and Fed rate decisions. The model's financial knowledge proved impressively comprehensive—it understood not only what death crosses were but also how to analyze their predictive value across different market environments.
The first indication of trouble came when Claude attempted to create a React-based dashboard visualization. The code looked remarkably sophisticated at first glance: properly structured components, appropriate imports from visualization libraries, thoughtfully organized data structures. A seasoned programmer might have mentally awarded points for the clean organizational structure and the thorough consideration of different market regimes and visualization types.
But then came the error message: "An error occurred while trying to run the generated artifact. 'Unexpected token, expected ";" (7:22)'."
When I pointed this out, Claude apologized profusely and attempted to fix the specific error, generating a new version of the code. Again, the result failed—this time with a different syntax error. After four attempts with similar errors, it became clear we were experiencing what software developers have come to call "vibe coding"—code produced through the natural language process of AI that looks plausible but contains subtle, disabling errors that reveal the AI's fundamental lack of understanding of execution context.
"You seem to keep getting the same error, a missing semicolon at 7:25," I noted after several failed attempts. "Can this be fixed another way?"
Claude pivoted to a simpler approach, abandoning the complex React components in favor of generating a more basic HTML and JavaScript dashboard. This too failed, with similar syntax errors. At this point, Claude suggested using the REPL tool (a JavaScript execution environment) to test smaller pieces of code first.
This was a sensible approach—one that human programmers would naturally take—but even the smaller code samples proved problematic. What should have been a 30-minute task for a competent web developer had now stretched beyond 90 minutes with nothing to show for it but a string of error messages.
The contrast was striking: the model could provide sophisticated financial analysis about how "the COVID-19 death cross in March 2020 was a significant anomaly, with all ETFs showing strong positive returns in the following 6 months," but couldn't properly create a functioning line graph to visualize this insight. It's like watching a brilliant market strategist with impeccable analytical skills struggle to operate PowerPoint.
Tables without Any Visualization: https://claude.site/artifacts/a17af92b-e6f3-489e-9464-c566b8ce0862
A Few Charts and Tables without the Interactive Visualizations (Missing Parts): https://claude.site/artifacts/084fd98b-302c-422a-a8e7-becbcc09fc44
More Missing Parts: https://claude.site/artifacts/bce0cddb-9b6e-4cdf-9ec9-66f4c2d27432
After multiple attempts and another half hour of trying while thousands of lines of code flowed down the screen, it became clear that I needed to abandon the interactive visualization approach entirely and move back in time to an earlier generation of code that Claude could handle (the 70's terminal pivod). In this final pivot that Claude ironically suggested and assured me to cool my frustration that it would handle without further error , Claude created a markdown document with text-based "charts" and data tables—essentially very early ASCII art representing what should have been dynamic visualizations with though at least an eloquent narrative . This primitive approach though at least worked, finally producing a deliverable that contained the analytical insights but lacked the visual sophistication originally requested.
The Working Real Possibility 2025 (Prepare Yourself): https://claude.site/artifacts/2f022bbc-fc68-4bcc-a951-603b607c655f
The markdown report itself highlighted the peculiar disconnect in Claude's capabilities and the current hype. It did include though (not to belittle Claude Sonnet 3.7), insightful statements like "fiscal-monetary policy coordination emerges as the single most powerful determinant of rate cut effectiveness (0.87 correlation)"—a nuanced insight that would impress financial analysts. But this sophisticated analysis was presented with crude text-based visualizations that wouldn't have looked out of place in a 1985 computer terminal.
Rate Cut Effectiveness Factors
(Correlation Coefficients)
Fiscal Coordination |████████████████████████████░░░| 0.87
|
Time to First Cut |███████████████████████▓▓▓▓▓▓▓| -0.79 This entire exercise consumed approximately 25,000 tokens (approximately words or long list of programmatic code) across multiple attempts, representing a significant both computational (cost) and human "time" investment for what ultimately amounted to a fallback solution that would not embarrass both of us (see linkk above, lol) . The time spent—nearly two hours—would perhap equal or even surpass what a mid-level web developer would require to build a functional dashboard from scratch and come to grips with the financial level of the equation but definitely not world class or what a Tableau programmer with a simple Excel spreadsheet of the data could far surpass in say 15 minutes.
It must be said though for a computer, the challenge Claude Sonnet 3.7 faced wasn't trivial and couldn't be even consided as something a computer could do say pre 2022 but on the other hand this was not superintelligence or even 'artfiical general intelligence' it was very clunky but albeit making it too the finish line, a little bruised and battered for both of us.
Interactive data visualization represents a good benchmark and complex intersection of design principles, programming logic, and data manipulation. The challenges require precise syntax, an understanding of browser rendering, and careful attention to user experience and a prompt are all levels of testabilty that needs to be improved, iterated and surpassed to match humans and especially human experts with these very real human/ai challenges. This level of technical execution stretches beyond simple code generation into the realm of software engineering—a domain where the gap between AI's natural language capabilities and execution skills becomes most apparent, especially in the trajectory and timeline we have yet to compete before getting rid of the Computer Science BSc. M Sc. and Ph.D as we know it - I'd say at least ten years for the Ph.D and the BSc. is still a worthy degree today if what Sonnet 3.7 produced today by any standard. To also mention thought, this is a great debugging tool for a Masters Student especially and a great help to all who have finished at least one degree in Computer Software development or have been coding for at least 5-10 years.
"Vibe coding" captures this phenomenon perfectly and it is easy to understand in this manner why most programmers are noticeably excited by the help possible. Claude does understand the basic concepts and can generate code that looks mostly right to human eyes, but does currently lacks the execution context and error correction proceddures to ensure that the code actually works and can be deployed easily by largely a practicing dilletant. It's like a chef who can describe a soufflé in exquisite detail but has never operated a restaurant with real consumers on the other end who also have real opinions about what has been produced and it's value towards 'paid' wider general consumption.
The more intriguing aspect is that models like Claude 3.7 Sonnet don't receive direct feedback from code execution during training. Unlike human programmers who learn through a continuous cycle of writing, testing, debugging, and refining, language models generate code based only on patterns they've observed in their training data. They can't "see" their errors in the way humans do and currently don't correct on the fly without producing the 'reams' of code going back to the drawing board mostly 'from the beginning' which is laborious for a human even to witness.
This limitation points to a potential evolutionary path for AI systems: integration with wider real time executionary environments that would allow models to test and debug their own code before presenting it to users. Imagine if Claude could run its visualization code in a sandboxed environment, identify the semicolon error at line 7, fix it, and present only working code to the user. Such a capability would dramatically improve the reliability of AI-generated technical artifacts. On various, industry standard for code evals, Sonnet 3.7 scores between 62-70% depending on who you ask (see tables in this article) I do completely agree here with these scores and benchmarking tool's score which is high by AI standards. In my book though, 62-70.0% is realistically still a C- student in terms of grades and getting things done. In school, this is a little more than barely passing not even close to 'Nobel prize level', but not to overlook things showing sincere effort, less than average but willing to improve and with their heart in the somewhat AI right place but for now, less than average in this score but definitely in it for the long game and willing to take instruction or as they say 'deep learning' and 'model training' school for a brand new and better version.
In this way we are in early innings and by human 'digital' instant time expectations, Sonnet is still also quite slow in producing this much code. A promising direction is the development of diffusion models specifically for code generation. Much like how image diffusion models iteratively refine outputs from blurry to focused and precise, code diffusion models could progressively improve generated code based on execution feedback and parallel processing to complete the refined finely tuned picture or coding artificant in more minimal humanly acceptable time. This approach might significantly reduce the wait time for in the end non-working code structures that mut be 'returned' to the drawing boards after prolonged and somewhat frustrating 'wait' times.
The time investment remains a critical concern. Spending ninety minutes to produce what ultimately amounted to a static document is far from the seamless experience promised by AI enthusiasts. For professionals looking to integrate tools like Claude into their workflows, this represents an unacceptable efficiency cost. Future improvements will need to address not just accuracy but speed—the ability to generate reliable code in seconds rather than requiring multiple iterations over extended periods.
Despite these limitations, it would be a mistake to dismiss the capabilities Claude did demonstrate as this would take a beginning programmer weeks to produce anything, professionals aside. The model also displayed remarkable domain knowledge across financial markets, technical analysis, and information towards investment strategy. It correctly identified some of the more important relationships between market indicators and monetary policy actions across four distinct market regimes. It generated insights that would be valuable to investment beginners, middle level casual browsers and even some of the professionals wanting a crib, even if the final presentation above felt more primitive and fell short of the latest in 21st century polished 'data visualization'.
This dichotomy—sophisticated attempts at largely unchecked or refereed finanical analysis paired with primitive cod execution—may be the defining characteristic of our current AI era that we are in - the early innings of a quickly changing game. These systems understand concepts at a dangerious level that can impress beginners, annoy domain experts and struggle with the precise execution that many professionals in both domain areas take for granted. AI's canf think and speak with the confidence experts but can't yet fully perform yet so the hype is not yet fully 'realized'.
What does this mean for the programmers and data visualization specialists and financial analysis whose jobs might someday be affected by these technologies? The current evidence suggests they can rest easier than many fear, at least for the next ten years though a misguided superior may be quit to let workers go in light of not knowing what they are really getting. While benchmarks from companies like Anthropic suggest their models can match or exceed human performance on certain programming tasks, the real-world application and results of a non professional in both of these areas with an AI reveals a more nuanced but 'realistic' picture. Models like Claude Sonnet 3.7 excel at coding tasks with well-defined parameters and immediate feedback but struggle with the open-ended, iterative problem-solving that characterizes professional software development and basic real world coding problems with a C+ at best and 'atta boy' for at least trying.
This isn't to suggest progress isn't coming. The pace of improvement in AI capabilities has been remarkable, with each new model generation demonstrating significant advances. The limitations observed today may be solved in the coming years or the next decade especially with very progressive advance. But for now, the gap between concept and execution—between understanding visualization principles and producing working visualizations— between human expert and AI - largely still remains unbridged.
Perhaps the most fitting visualization of AI's current capabilities is this very contrast: a system that can analyze market data with PhD-level confidene towards it's insights but struggles to create a simple bar chart and freely and happy admits that it had hallucinated a couple of the sources when pressed. In this gap lies the true state of artificial intelligence in early 2025—very impressive on the surface, advancing rapidly with capabilitie, but still distinctly on a lower tier and non-human in its patterns of both larger strengths and greater limitations.
As we look toward the future development of these systems, the key question isn't whether they'll eventually clear these hurdles toward even greater heights—they almost certainly will—but rather what new capabilities and limitations will emerge when they do? The computational gap will close, but in doing so, it will likely reveal new frontiers where human creativity and judgment remain indispensable.
For now, though, if you need a sophisticated analysis of death crosses and their relationship to Federal Reserve policy, Claude 3.7 Sonnet can provide remarkable insights. Just don't ask it to show you a chart on your first attempt at having it run the code. And while AI autonomous agents are the hot new kids on the block for 2025, Gemini 2.5 pro at the top of the class for this group is still getting things wrong at least 25% of the time (see diagram below) and Sonnet squarely in the middle at a worthy 62% , still better than it's venerable cohorts but still only running at a slightly better than average C-. In other words, the whole top class still has at least a few things to still learn before being promoted to the next grade, especially with our next generation of agents. Give it at least till the middle of 2026 for the whole or at least most of this class to top this one.
Full Original Prompt, Sonnet 3.7 Source Code and All Attempts
https://claude.ai/share/0af69015-4025-4a14-ae9b-0012da235686
Outside Review of Claude Sonnet 3.7 and Claude Code
https://www.helicone.ai/blog/claude-3.7-benchmarks-and-examples
#ClaudeSonnet3.7 #AnthropicCoding #Claude3.7Coding #AICoding #AIProgramming #AICoding2025 #FutureofWork