Singularitarianism | Philosophy of artificial intelligence
AI alignment
In the field of artificial intelligence (AI), AI alignment research aims to steer AI systems towards their designers’ intended goals and interests. An aligned AI system advances the intended objective; a misaligned AI system is competent at advancing some objective, but not the intended one. AI systems can be challenging to align and misaligned systems can malfunction or cause harm. It can be difficult for AI designers to specify the full range of desired and undesired behaviors. Therefore, they use easy-to-specify proxy goals that omit some desired constraints. However, AI systems exploit the resulting loopholes. As a result, they accomplish their proxy goals efficiently but in unintended, sometimes harmful ways (reward hacking). AI systems can also develop unwanted instrumental behaviors such as seeking power, as this helps them achieve their given goals. Furthermore, they can develop emergent goals that may be hard to detect before the system is deployed, facing new situations and data distributions. These problems affect existing commercial systems such as robots, language models, autonomous vehicles, and social media recommendation engines. However, more powerful future systems may be more severely affected since these problems partially result from high capability. The AI research community and the United Nations have called for technical research and policy solutions to ensure that AI systems are aligned with human values. AI alignment is a subfield of AI safety, the study of building safe AI systems. Other subfields of AI safety include robustness, monitoring, and capability control. Research challenges in alignment include instilling complex values in AI, developing honest AI, scalable oversight, auditing and interpreting AI models, as well as preventing emergent AI behaviors like power-seeking. Alignment research has connections to interpretability research, robustness, anomaly detection, calibrated uncertainty, formal verification, preference learning, safety-critical engineering, game theory, algorithmic fairness, and the social sciences, among others. (Wikipedia).
