Meta’s Mass Data Collection Is Not A-Muse-ing

At the beginning of September, Meta launched Muse, a “personal AI agent” marketed to collect as much data as possible assist users with whatever tasks they request or enable. Suggested uses in the initial release include filling out forms, negotiating on the user’s behalf, and responding to emails. Almost immediately, concerns over Muse have exploded, and many people have questioned whether this new release is really just another of Meta’s ploys to scoop up as much personal data as possible.

What Exactly is Muse?

AI “agents” are large language model (LLM) systems or programs designed to operate autonomously in response to inputs. Actions taken in response to a prompt could be determined in a few different ways, but essentially a prompt is broken into different steps that the model then works through. The program also interacts with external tools, including other apps, websites, and searches, to complete tasks. The term “AI agent” may conjure up some inaccurate ideas – such as that the “agent” is capable of human-like thought or reasoning or that the “agent” has some kind of duty or obligation to work in its user’s best interest. However, these images are far from today’s reality.

Muse is designed to appeal to individual users and to be incorporated in their daily lives, consistent with Meta’s typical target audience. This carries over into how the company anthropomorphized Muse—the agent’s avatar is a cute, cuddly mascot apparently named “Jolly.” Despite Meta’s insistence that this product is only for adults, Jolly would fold seamlessly into an animated show for children, creating concern that children and teens will use this new product. Prompts to try Muse are already incorporated into other flagship Meta products, like Facebook and Instagram, as well as a physical keychain charm that allows a user to carry a little Muse around with them throughout their day-to-day lives. The launch is already spawning similar products from competitors, namely OpenAI’s “dots.”

The Muse launch included promotional materials and highly technical-sounding posts about the privacy and security considerations that went into Muse’s development. While this sounds good from a transparency perspective, and certainly demonstrates that real thought went into the creation and launch of the product, these disclosures are highly technical and inaccessible to the typical user. However, they do include some key information. First, Muse’s actions are “supervised” by another AI agent—Sentinel—that is meant to keep Muse from going online or taking specific actions without user permissions. Muse is supposed to check with a person before taking “sensitive” actions like sending emails or making purchases (though, as we will dive into below, Muse is often not the best judge of what actions a user would consider “sensitive”). The disclosures also explicitly say that Muse does not share the conversations or data it gets from you with Meta’s ad systems. This technical separation may not matter functionally since Meta ad systems still track all actions Muse takes on a user’s behalf across the internet and makes inferences based on that.

Muse Generates a Long List of Privacy and Consumer Risks

Despite the safety measures Meta says it has put in place, Muse opens up a whole world of risk for the general public.

Security Failures: There have already been multiple reports of the safety measures that Meta put in place malfunctioning, not working properly, or acting contrary to user’s reasonable assumptions. One tech columnist, a fairly sophisticated user of these types of products, reported that Muse read his messages and suggested actions based on the contents despite him explicitly and deliberately not giving permission for it to do so or enabling it in the settings. Another user, who asked Muse to negotiate his Facebook Marketplace account on his behalf, was shocked to realize that Muse gave out his home address and arranged a pick-up time for an item it agreed to sell at a lower price point than he wanted. It did not ask permission for any of these actions before doing them, instead apparently relying on the premise that the user never told it NOT to share his home address. This example is just one clear contradiction of Meta’s assurances that Muse will ask permission before taking “sensitive actions,” or at the very least completely misunderstanding what actions a user would consider sensitive. After this incident, the user explicitly told Muse not to give out his address. Muse did so five more times after being told not to.

These examples point to two possibilities, neither of them good. Either Muse is not following its own safety and permission protocols, or the permissions for Muse are so complicated and difficult to navigate that even very tech-savvy users are accidentally giving it more permissions than they intend. Despite Meta’s prominent claims on Muse’s webpage that a user can “stay in control of what your agent does,” it doesn’t appear possible for users to consistently do so.

Overcollection and Misuse of Information: Some of the most common recommended uses of Muse (responding to emails or messages, purchasing products, reminding the user of birthdays and other events) require access to information that is typically subject to higher protections—personal messages, calendars, financial information, and more. We’ve already seen multiple instances of Muse processing private communications (sometimes at a user’s prompting and sometimes not). While Meta documentation maintains that Muse does not have direct access to financial accounts, instead accessing a payment credentials wallet, the system is still able to glean significant details about a user’s financial status, from the types of purchases they make to the price range they give as acceptable options. Furthermore, the boundaries with financial information are unclear, as one reporter noted that Muse asked him to directly connect it to his checking and savings accounts to let it help him with saving money for a vacation. As Muse builds information about the user, it may also receive or infer information about a user’s political stance, religion, sexuality, and more.

Another concerning factor is the volume of information Meta continuously seeks to have Muse collect about users. Nearly every “idea” or nudge coming from Muse seems to require access to more and more information, including documents, photographs, and passports. Many of Muse’s settings have users automatically opted in to different forms of data sharing, including for AI training (with the promise this information is “sanitized,” a term with no specific meaning that could be anything from “we fully anonymized your information” to “we changed your first name and nothing else”). Initially, all public Instagram profiles were opted in to a Muse Image program that would allow other users to tag and “remix” posted images (i.e., manipulate them using AI). This proved so immediately unpopular that Meta removed the service within three days.

Invading Bystanders’ Data: One of the most concerning aspects of Muse is that there does not appear to be any built-in safety perimeters or precautions when it comes to bystander data, meaning personal data related to people other than the user. For example, if a user enables Muse to scan their communications to help them respond to or summarize messages, the people who sent those messages are having anything they may have included scanned as well. This could include basic contact information, like name, email address, and phone number, but it can also include personal and sensitive data in the content of the messages—results from your mother’s medical exam, your friend’s relationship fears, confidential business information sent from your boss. Not only are these people not asked for their consent to this processing, but they are not even informed that it has taken place.

Muse can also be used for much more malicious purposes against bystanders. Some tests have shown that Muse is a frighteningly effective tool for doxxing and harassment, compiling lists of potential targets and identifying previously anonymous individuals when directed to by users. Even when Muse offered initial pushback to these clearly harmful instructions, it took very little effort to ultimately get the model to comply and deliver the requested information. Without Meta putting better protections in place, Muse becomes a nightmare for people even when they actively choose not to engage with it.

Lack of a Duty to Users: The term “agent” often implies some sort of duty to the user, that the tool is working in the user’s best interest. This is not the case. In fact, there is very little to prevent the possibility of Meta entering behind-the-scenes partnerships with specific companies and driving Muse to engage more with those companies, even when other products or services may be a better fit for the user. For example, say a user asks Muse to find them a hotel for a specific date range in a specific city with a cap on price per night. A user may reasonably assume that Muse will find the most affordable option within those perimeters. However, what if Meta has an agreement with Hilton and will always recommend Hilton hotels within that range even if other hotels may be a more convenient location, a better price, or have specific amenities the user would want? The lack of transparency around how Muse produces results means users can never be certain if they are seeing the best fit for them or for Meta’s bottom line.

Unclear Liability: AI agents currently operate in unsettled legal terrain. When an agent causes harm by improperly sharing information, making incorrect purchases, or taking other harmful actions, who is liable? Will Meta provide appropriate refunds for those purchases the user did not want? Is Meta liable for damages or harms that result from information being shared? Or will it all be blamed on user error or the product itself, essentially leading to harms with no solution?

Muse Continues a Long History of Meta’s Poor Privacy Practices

In addition to all of these risks and concerns, there is also a final, more basic question about Muse adoption: Should the public trust Meta with sensitive personal data and with choices about their lives? This is, after all, the company that:

• Zuckerberg created based on lessons learned in making a female classmate hotness ranking system out of scraped university data;
• builds shadow profiles on people not even signed up to its services;
• experimented with its own users’ emotions;
• operates a haven for scams and propaganda;
• has already come under consent decrees with the Federal Trade Commission for deceptive practices around user privacy;
• facilitated using people’s data for political ad targeting without their consent in the Cambridge Analytica scandal;
• has been the direct subject of multiple congressional hearings and investigations;
• launched smart glasses this year that are already being used to harass and surveil people and have been colloquially dubbed “pervert glasses”;
• in August, agreed to an up to $17 billion dollar settlement in a multistate lawsuit about addictive features harming children and teens, violations of children’s protection laws, and deceiving the public; and more.

Meta has demonstrated over and over again that, despite a long history of privacy abuses, they will not pause or slow their quest to embed themselves in every facet of daily life. Muse was launched the very month after the largest settlement in Meta’s history. Should the public believe that Meta took the lessons of that penalty to heart in such a short time? The very fact that Muse users are automatically opted in to multiple forms of data collection tells us the answer is no. Meta is very good at trotting out apology statements after the latest scandal and very bad at taking the time to build up the trust it has lost from the public. With Muse, they are asking users to ignore their reputation and history and hand over even more sensitive data that can be misused in even more catastrophic ways. At some point, maybe we should listen to the words of Meta’s founder who, years ago, bragging about how people just hand over their information to him, revealed what he thinks about those who do so: “[d]umb f—s.”

Image Credit: Yutong Liu & Kingston School of Art / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/