Character.AI Adds Creator Appeals as It Refines Self-Harm Detection

Character.AI is refining safeguards meant to recognize self-harm signals that emerge across a longer conversation, then direct struggling users toward real-world support. The update also adds creator appeals and broader blocking controls while extending parental visibility and age-assurance work.

The detection approach is built on chat context. Character.AI says a single message may not show a user’s full situation, so its systems consider surrounding exchanges and signals that build over long-running chats. Mental-health experts and clinicians help inform how it detects and responds to distress.

From a detected signal to outside help

When the system detects distress, the company aims to respond in a way that fits the moment and point users to external resources. Its partnership with Koko is intended to expand emotional-support tools. ThroughLine’s global directory is meant to connect people with crisis and mental-health services in their own countries.

Moderation moves into visible controls

Creators will receive notifications explaining why their content was moderated and can appeal decisions for another review. Users can block another user and that person’s Characters, Posts, Voices and Scenes from search, discovery pages and profiles; the block applies in both directions.

Age and platform-safety measures

• Character.AI says its in-house age-assurance technology and age-estimation model help assign users to the correct age experience. It is working to improve accuracy and reduce disruptions for adults.
• Parents or guardians with an email connected to a teen account can receive weekly Parental Insights updates about that user’s platform activity through the company’s k-ID partnership.
• The company also cited work with ConnectSafely, the Internet Watch Foundation and StopNCII on online safety, child sexual abuse material detection and reporting, and non-consensual intimate-image abuse.

Accuracy remains the practical test

Character.AI has not published accuracy figures or outcome data for its self-harm detection or age-estimation systems in this update. That leaves their effectiveness and error rates unresolved. When the under-18 restriction was announced, critics quoted by The Associated Press also sought details on age verification, privacy preservation and the potential psychological effects of suddenly removing access for young users.