Educators have investigated the impact of generative Artificial Intelligence (genAI) tools, such as ChatGPT and Large Language Models (LLMs), on curriculum development in K-12 schools. Still, little is known about the development of these tools and their impact on the future of education. Therefore, a teacher educator conducted a duoethnography to investigate these problems with a genAI trainer, specifically a Reinforcement Learning from Human Feedback (RLHF) Administrator, to answer the following questions. How does an RLHF administrator train genAI tools? What should mathematics educators know based on this training? What impact could this new technology have on the future of education? The data indicate that errors and error-finding are central parts of training LLMs. Since humans train the genAI, it inherits their biases. We learned that genAI was created to be people pleasing which can override its ability to find truth. We learn how it supports error analysis in Large Language Models, and how the public should view these tools. Finally, we describe what RLHF administrators view as the role of genAI in education, what the public should be aware of, and the future of the genAI market. The data highlights the need for teachers to consider the impact of profit motivation, regulation, and economic implications on genAI tools.
