AZ and KA were stunned when they found out that their photos have been edited into an indecent form of content using artificial intelligence by their own highschool friends. AZ mentioned that not only did their photos get edited, but other friends of the perpetrator have also fallen victim to these same acts. In reality those are not real photos, rather they are pictures that have been engineered using Artificial Intelligence (AI) or what is usually called deepfakes.
The perpetrator committed these acts with the goal of making AZ and KA as a sort of joke material for their friend group. Both girls at that point in time were 16 years old, originally they were told by their friend that saw the perpetrator showing off those photos and were offended and feared for her safety, but she herself didn’t know where to report such a case. She also felt that if she reported it, this will only result in a boomerang effect for herself and her friends.
AZ and KA finally decided to confront the perpetrator so that he may admit his mistakes and apologize immediately. Of course, after this incident AZ and KA’s friend group have decided to distance themselves from the perpetrator to maintain safety.
The violence that AZ and KA experienced is one of many examples of Online Gender-Based Violence (KBGO) by using deepfakes. The perpetrator also is still running free without any punishment whatsoever. Mirroring this case, we can see how the misuse of artificial intelligence keeps increasing victims of such acts. The question is, what exactly is the root cause with deepfakes?.
What is a Deepfake?
BBC Science Focus in an article titled “The 25 most powerful ideas of the 21st century (so far), picked by the world’s top thinkers” puts artificial intelligence at the 9th ranking. This article states several scientific breakthroughs that have changed our world since the beginning of this century.
Technological developments race alongside current development, no wonder something such as these gives us new challenges. Artificial intelligence shapes a new spectrum that has the potential to both help humanity and destroy human lives at the extreme end of it.
The United States Government Accountability Office stated that deepfakes are videos, photos, or audio recordings that appear to be real but have been manipulated using artificial intelligence. This technology is able to change and manipulate facial expressions and even synthesize their face and their words. Deepfakes can picture someone that appears to be saying and doing things that they have never said or done.
This technology is often misused and frequently appears in the form of hate speech, political propaganda, and pornography. One of the most prevalent forms of deepfake-related crime is nudification, which involves using computer technology to manipulate photos or videos to make a person appear nude.
Nudification threatens people of all ages, including children. Based on a joint study by UNICEF, ECPAT, and INTERPOL across 11 countries, at least 1.2 million cases were reported by children stating that their photos had been manipulated into nudified deepfakes.
Accessibility of Deepfake Use on Social Media and Illegal Websites
The threat of deepfakes has entered our shared digital public space. Social media platforms serve as spaces where individuals express their identities and shape their digital self-image. However, the rise of deepfake technology has the potential to damage a person’s self-representation, making the digital environment increasingly vulnerable to such forms of abuse and exploitation.
The integration of artificial intelligence into social media platforms has created new opportunities for nudification-related crimes. The presence of Grok, an AI assistant on X, was initially intended to summarize complex information, provide contextual explanations with comprehensive references, answer questions directly, and enable fast image generation. Even so, its relatively uncontrolled image generation feature has instead made it easier for perpetrators to carry out nudification-related offenses.
According to the Center for Countering Digital Hate (CCDH), the artificial intelligence developed by Elon Musk is estimated to have generated around 3 million sexually explicit images, 23,000 of which depict children. This occurred following the launch of a new image-editing feature in Grok.
Perpetrators of nudification can carry out their actions with alarming ease. In open comment sections, they can input prompts to manipulate images into non-consensual nudified content without the photo owner’s permission. Within a short time, Grok responds to these prompts by generating deepfake results that are then displayed publicly in the comment section.
The digital space has increasingly become unsafe. Users’ photos and videos are now primary targets for perpetrators carrying out deepfake attacks. The threat lies in the ease of access and the broad capabilities of these technologies. Users of X automatically have access to features like Grok without complicated registration processes, which increases the risk of misuse. However, X has introduced certain limitations through a subscription (premium) feature, meaning users must pay to access Grok.
In addition to Grok, there are many deepfake tools specifically designed for nudification that are used by perpetrators of abuse. We tested one such nudification tool that can be easily accessed through a website.
We strongly condemn any act of using artificial intelligence to perform nudification without the consent of the individuals involved. Our testing was conducted using fictional two-dimensional human characters that do not represent any real person, and was carried out solely to assess how easily these websites can be accessed in Indonesia.
To examine how easy the nudification process is, we conducted tests on two deepfake tools available on the internet. The testing was carried out based on several aspects, does it require the use of a Virtual Private Network(VPN)? Is the website blocked by Internet Positif? Are the users required to log in? Are users required to pay for access to the tools?
The website undress.app can be accessed without using a VPN and does not trigger an Internet Positif warning. Internet Positif is a policy that blocks certain websites deemed illegal or inappropriate. On this platform, users are required to log in and purchase tokens to use the tool. This makes the tool slightly less accessible, but anyone who is able to pay for the tokens can still easily gain access. The level of realism is quite high and can be convincing, especially for those who are not careful in identifying deepfake manipulations.
Furthermore, the website clothoff.net can be accessed without using a VPN and is not blocked by Indonesia’s internet filtering system. What is particularly concerning is that users do not need to log in and are even given free tokens to try the service. This allows perpetrators to generate multiple images without authentication, making them more difficult to trace.
Although the level of realism is not as high as some other platforms, the tool can still deceive individuals who are not familiar with detecting deepfake manipulations. These findings indicate that nudification tools are highly accessible and can be easily used by perpetrators to carry out their actions.
This highlights the importance of strengthening preventive regulations through cross-sector collaboration. Including cooperation between the government, internet service providers, and filtering organizations such as SAFEnet in efforts to block and monitor digital platforms that have the potential to be misused.
Lack of Regulation and Strict Control
Regulations on artificial intelligence in Indonesia remain limited and are often considered too general. The Electronic Information and Transactions Law (UU ITE), specifically Articles 27 and 27A, addresses the prohibition of distributing electronic content that violates decency or constitutes defamation. However, these provisions regulate the content itself rather than the use or development of artificial intelligence.
In addition, there is UU No. 12 of 2022 on Sexual Violence Crimes (UU TPKS), which is related to pornography and online gender-based violence. However, these provisions cannot yet be fully relied upon as a comprehensive primary tool to address the misuse of artificial intelligence, particularly in cases of deepfakes.
A member of the Committee for Corporate Responsibility of Digital Platforms for Quality Journalism (KTP2JB) and a human rights expert, Damar Juniarto, stated that existing laws are still insufficient because artificial intelligence itself has not been specifically regulated. According to him, AI is not designed to be difficult to use or limited to specific functions. As a result, this technology is easily accessible, simple to operate, and has minimal built-in restrictions.
Damar explained that the root of the problem lies in the fundamental nature of this technology as a general-purpose technology. This differs from other technologies designed for specific functions, such as a calculator for computation; generative artificial intelligence is created without rigid functional boundaries, making it far more complex and open-ended in its use.
Moreover, artificial intelligence is now highly accessible and can be operated by almost anyone. It is very flexible and comes with minimal technical barriers compared to earlier technologies. This ease of access becomes a double-edged sword. On one hand, AI drives human innovation; on the other, it creates wide opportunities for misuse due to the lack of inherent limitations or built-in constraints.
He also highlighted the absence of a Human Rights Impact Assessment (HRIA) in the development of artificial intelligence. HRIA is a strategic auditing process conducted to understand and mitigate the potential negative risks of a technology.
Currently, The United Nations (UN) promotes human rights standards for both AI and non-AI technology companies through the framework of the United Nations Guiding Principles on Business and Human Rights (UNGP’s). These standards are applied more specifically through the Business and Human Rights in Technology (B-Tech) project. The UNGPs are comparable to an Environmental Impact Assessment (Amdal), but in the context of human rights and their relation to technology.
The focus of HRIA standards is to shift from merely “technology ethics” to concrete legal and operational “responsibility” to prevent, mitigate, and remedy the negative impacts of technology on human rights.
Artificial intelligence can be used across a wide range of tasks closely tied to everyday human activities. Companies often, whether intentionally or unintentionally, overlook certain risks associated with how their products are used by users.
For example, a company develops an artificial intelligence system to recognize employees’ faces for attendance tracking in the office. Without a Human Rights Impact Assessment (HRIA), this facial data could be sold to illegal online lending syndicates. The data could then be used to create new loan accounts under the employee’s name. The presence of HRIA encourages companies to anticipate worst-case scenarios in the use of artificial intelligence from the very beginning of its development.
Chair of the Resource Center at The National Commission Against Violence against Women (Komnas Perempuan), Chatarina Pancer, explained that deepfake is a sophisticated form of artificial intelligence manipulation that is difficult for the general public to detect. Although there are regulations such as the Personal Data Protection Law (PDP) and the Criminal Code (KUHP) addressing identity falsification, law enforcement remains slow compared to the rapid development of technology.
Assistant Coordinator for Information Technology at Komnas Perempuan, Aldi Pratama, described challenges related to Electronic System Operators (PSE) that do not have local offices in Indonesia, making them difficult to regulate. He emphasized the need for regulations to ensure that artificial intelligence cannot process individuals’ facial data without consent, as well as the importance of social media accountability, including stricter control over access for minors.
Komnas Perempuan collaborates with the Southeast Asia Freedom of Expression Network (SAFEnet), a digital rights advocacy organization, to facilitate content takedowns when victims file reports, as well as to coordinate with Komdigi or platforms such as Meta. To minimize risks, the public, especially young people, is encouraged to understand how AI works and its potential dangers. This effort is further strengthened by empowering journalist communities to disseminate information about the risks of deepfakes, which is considered crucial in building a supportive system for victims.
The Annual Report 2024 (Catahu) by Komnas Perempuan outlines various forms of online gender-based violence (KBGO), including online threats, privacy violations, and the distribution of harmful content. It is important to understand that a single reported case may involve more than one type of offense. Komnas Perempuan also emphasizes that eliminating AI is not a feasible solution, as it has become an essential tool in many areas of work. Instead, the focus should be on ethical use and stronger government oversight.
Deepfake Cases in Indonesia
Recently, deepfake cases have begun to emerge in Indonesia. In 2022, a fake video of the celebrity Nagita Slavina circulated, created using deepfake technology. Furthermore in 2023, 16 junior high school (SMP) students in Malang and 17 women, consisting of the perpetrator’s close friends, colleagues, and former partners, became victims of deepfake content distribution. The survivors’ personal photos were manipulated in a vulgar manner using deepfake technology and shared without consent.
The Association of Legal Aid Institutions for Women and Children (LBH APIK), in its LBH APIK Jakarta Annual Report 2022, recorded 49 cases of non-consensual dissemination of intimate content and 14 cases of reputational damage involving images, writings, or videos containing explicit or indecent material.
We also distributed an online form-based survey, titled The Impact of Artificial Intelligence (AI) Misuse in Non-Consensual Pornographic Content in Indonesia. Twenty-six out of 40 respondents stated that they had witnessed such misuse. However, the majority of these respondents chose to ignore it or handle it privately. As they did not know which authorities to report to or sought to protect their own dignity and that of the victims.
Most respondents feel that the current regulations in Indonesia are not sufficient to protect victims from the misuse of artificial intelligence in generating pornographic content.
Chatarina Pancer stated that the primary impact is psychological devastation for victims, ranging from panic and depression to withdrawal from their social environment. Victims may also experience economic losses such as losing their jobs, physical health issues caused by stress (metabolic disorders), and even self-harm.
Findings from cases and data indicating low reporting rates, reveal a gap between available protection and access to handling mechanisms. Victims often choose to remain silent because they do not know where to report or feel that doing so may pose risks to themselves. Addressing deepfake cases requires a response that is swift, well-coordinated, and sensitive to the victims’ conditions. The presence of SAFEnet and Komdigi is therefore crucial in providing both technical assistance and support services for victims.
SAFEnet’s Social and Cyber Psychology Researcher, Wida Arioka, explained that the organization plays a direct role in assisting victims of online gender-based violence. Through collaboration with various digital platforms such as Meta, Google, TikTok, and Telegram, SAFEnet helps facilitate the reporting process as well as the handling (takedown) of harmful content.
In addition to technical handling, SAFEnet also provides psychological support services through e-counseling, an online counseling session between victims and professional practitioners. This approach is important given that deepfake cases can have serious impacts on the mental well-being of victims.
Komdigi provides a reporting channel that the public can use to report various forms of negative content. Ranging from websites, links (URLs), social media accounts, mobile applications, to software that violates legal regulations. Users simply need to access the aduankonten.id page and then register through the available form.
After that, the reporter can upload the link and screenshots of the reported content, along with the reason for the report. The reporter can monitor the progress of the case through the system that has been provided. More detailed information about this service mechanism can also be accessed through the FAQ feature of the website.
