Artificial intelligence is more complex than any technology wave before it. Today, writing code and scaling infrastructure seem easy, but that simplicity hides many hidden decisions. The real challenges now involve judgment, coordination, and systems thinking. A new role called the “AI engineer” is emerging. This person focuses on applying large language models (LLMs) using APIs or open-source tools to build, test, and improve AI systems, rather than training models from scratch.

As this role develops, engineers are moving away from the core programming skills they used to rely on. Instead, they need new skills suited to a world where AI generates initial code drafts. One of the most important shifts is how we think about testing and measuring AI systems. Instead of only writing tests for software, engineers now focus on evaluation — continuously checking how well models perform and making improvements based on those results.

Evaluation: The New Standard in AI Development

Jeff Boudier from Hugging Face compares evaluation to what continuous integration (CI) was for traditional software. In the past, CI helped developers automate testing and ensure code quality. Now, evaluation is doing the same for AI. It involves creating metrics, running tests, and swapping out models when better ones become available. Hugging Face’s tools make this process seamless. They provide libraries and platforms that let developers assess hundreds of models quickly, compare their performance, and keep improving their AI systems.

This focus on evaluation is changing how companies develop AI. Instead of just picking a model and deploying it, businesses are building systems that can constantly measure and adapt. Experts say this makes AI development more systematic and reliable. It also emphasizes the importance of creating custom evaluation datasets that reflect real user conversations, ensuring AI models are tested against scenarios that matter for their specific use cases.

Designing for Change: The Need for Adaptability

With AI models evolving rapidly, adaptability has become a core skill. Unlike past software that changed slowly over months or years, AI systems can shift in weeks or even days. New models, API updates, and performance benchmarks appear constantly. Engineers must build flexible systems that can swap out components easily without disruption.

Barun Singh from Andela points out that this ability to adapt quickly is now one of the most valuable skills in software engineering. It’s not just about learning new tools but about creating workflows that can handle continuous change. Building boundaries through testing and good workflows helps catch mistakes early and keep systems stable. In this environment, being able to think both high-level and detail-oriented at the same time is crucial.

Managing Risks and Ensuring Ethical AI

Another vital skill is de-risking. Engineers are now taking on responsibilities similar to compliance officers. They need to ensure data sources are trustworthy, models are transparent, and pipelines are secure. As regulators start asking who is responsible when AI causes harm, transparency becomes an engineering requirement. Companies are expected to show clear lineage and governance over their AI systems.

Jessica Li Gebert from Neudata describes data as a “treasure trove” but warns many companies don’t know how to unlock its value safely. Engineers who can implement governance measures and manage data risks will be essential for AI to be adopted responsibly. Most enterprises still see sharing data with AI developers as risky, but those with expertise in both data systems and risk management will lead the way.

Building Systems for the Future of AI

Over the last two decades, companies competed to improve software creation through better tools like CI pipelines and cloud platforms. Now, the focus shifts to the systems that bring similar rigor to AI. Engineers who develop evaluation loops, model registries, and governance frameworks are not just keeping up with innovation—they are shaping how AI integrates into everyday business.

Just as CI made software more reliable and predictable, these new systems will make AI behavior measurable and improvable. They will help organizations deploy AI confidently, knowing they can monitor, evaluate, and adapt systems as needed. Ultimately, this new wave of engineering will determine how effectively AI becomes part of enterprise workflows and daily operations.

Inspired by

• https://www.infoworld.com/article/4083484/the-hidden-skills-behind-the-ai-engineer.html

Sources

• gmpg.org
• huggingface.co
• huggingface.co
• parlance-labs.com
• andela.com