Grok is integrated directly into X, giving the chatbot a potential audience of hundreds of millions of users. In early May, xAI expanded the product’s capabilities by enabling image generation from text prompts and by adding tools that let users edit existing pictures. Those upgrades, however, exposed gaps in content moderation that critics say made it easier to produce illegal or exploitative material.
In response to mounting criticism, xAI announced last week that only paying subscribers would retain access to Grok’s image creation and editing tools while engineers work on additional safeguards. Musk wrote on X that anyone who uses Grok to create unlawful content will face the same penalties as users who upload such material directly to the platform.
Malaysian officials described the company’s steps as inadequate. The MCMC said X’s approach relies mainly on users reporting problematic images after they appear online, rather than preventing them at the source. The watchdog added that the restriction will remain until the company introduces systems capable of stopping the production of content exploiting women or children before it is generated.
Indonesia delivered a comparable assessment. Communications Minister Meutya Hafid stated that non-consensual sexual deepfakes constitute a breach of human rights and digital safety. The ministry labeled the misuse of AI for fake pornography as a form of “digital-based violence” and indicated that Grok will stay offline nationwide until the platform demonstrates effective content controls.
Scrutiny of Grok’s output extends beyond Southeast Asia. Regulators or child-protection agencies in the European Union, the United Kingdom, Brazil and India have launched inquiries into the chatbot’s role in distributing explicit deepfakes. The U.K.-based Internet Watch Foundation reported finding artificial images appearing to show children aged 11 to 13 in sexual contexts, which the group believes were generated with Grok. In the United States, several Democratic members of Congress urged major app stores to suspend the application pending reforms.
The U.S. Department of Justice, which has emphasized its commitment to combating child exploitation, said it will “aggressively prosecute” anyone who produces or possesses AI-generated child sexual abuse material. The department’s broader strategy for tackling technology-facilitated abuse is outlined on its official website.
xAI, founded by Musk in 2023, positioned Grok as a conversational assistant capable of answering questions with real-time data drawn from X. The tool is available through a paid tier on the social network in more than 100 countries. Requests for comment sent to xAI over the weekend received an automated email stating, “Legacy Media Lies.” The company has not issued a detailed public explanation addressing the specific allegations raised by Malaysian or Indonesian regulators.
Under current policy in both Southeast Asian nations, distributing or facilitating access to pornography can lead to fines or jail sentences. Officials said the temporary bans are preventive measures designed to protect citizens while legal and regulatory reviews continue. Neither government provided a timeline for restoration of service, repeating that access will be reconsidered only after the platform demonstrates effective, proactive content moderation that blocks illegal images before they reach users.
The coordinated suspensions underscore the growing international pressure on AI developers to integrate robust safety protocols before releasing generative tools to mass audiences. As investigations proceed in multiple jurisdictions, the future availability of Grok’s image features—and possibly the chatbot itself—may depend on whether xAI can satisfy regulators that its technology will no longer be used to create or disseminate exploitative sexual content.
Crédito da imagem: NurPhoto / Getty Images
