The Department of Homeland Security is considering hiring private companies to analyze public social media for warning signs of extremist violence, spurring debate within the agency over how to monitor for such threats while protecting Americans’ civil liberties.

The effort, which remains under discussion and hasn’t received approval or funding, would involve sifting through large flows of internet traffic to help identify online narratives that might provide leads on developing attacks, whether from home or abroad.

The initiative comes after the nation’s intelligence community failed to sufficiently identify and share signs of the threats that led to the assault on the U.S. Capitol by a mob of Donald Trump supporters on Jan. 6.

John Cohen, a top DHS official, is spearheading the project, which he describes as part of an upgrade to the department’s capabilities in social-media analysis. Marshaling the expertise of outside companies and ramping up internal capabilities are central to that effort, he said.

“What we’re talking about now is dramatically expanding our focus,” Mr. Cohen said in an interview.

Mr. Cohen’s push has sparked internal debates in DHS and elsewhere in the Biden administration over longstanding tensions between civil liberties and security efforts. Some officials in the agency and the White House worry about governmental overreach, say people familiar with the deliberations, a concern that civil-liberties advocates share.

People familiar with DHS’s effort say the department needs to improve its capabilities after its intelligence arm failed to report social-media posts ahead of the Capitol attack, a lapse that has since been the subject of congressional probes.

The basis of DHS’s legal authority for the proposed expanded effort remains murky, said one person involved in the discussions, in part because “the definition of what is public is not necessarily settled.”

Adam Schwartz, an attorney at the Electronic Frontier Foundation, a nonprofit supporting digital rights, said the organization is wary of any government effort to “gobble up” more social-media data. That information could potentially be stored forever and later mined to find leads on crimes, he said.

“We do not support an expansion of social-media surveillance in the name of stopping the next attack on the Capitol,” Mr. Schwartz said.

U.S. intelligence agencies are seeking to address their failures to sufficiently respond to threats preceding the Jan. 6 storming of the U.S. Capitol.

Photo: Shafkat Anowar/Associated Press

The government currently tracks broad swaths of publicly available information, but it faces constraints on what it can gather from social media. While policies vary across the government, some agencies, such as the Federal Bureau of Investigation, often need evidence suggesting criminal activity to drill deep into online behavior.

On Friday, DHS extended a national terrorism advisory that it has issued twice since Jan. 6, highlighting concerns about violent extremism on social media. The advisory said foreign and domestic extremists continue to call for violence against government facilities and law enforcement, among others.

“This includes information regarding the use of improvised explosive devices and small arms,” the advisory said.

Balancing efforts to head off threats with free-speech protections can be difficult. Last year, the DHS intelligence office prohibited the reporting of veiled or indirect threats found on social media following backlash over its collection of intelligence surrounding demonstrations in Portland, Ore. As a result, the day before the Capitol attack, the office told law-enforcement agencies that it had “nothing significant to report.”

A DHS spokesperson said the department and White House are working together to ensure civil rights and liberties are upheld, though one official said the White House has pushed back on some of the department’s efforts.

Expectations are low that the Jan. 6 commission will be much more than a setting for a partisan food fight. But as WSJ’s Gerald F. Seib explains, even partisan food fights can bring to the surface important information, as might ongoing court cases. Photo illustration: Todd Johnson The Wall Street Journal Interactive Edition

DHS’s intelligence office detected online chatter threatening to disrupt President Biden’s inauguration on Jan. 20, prompting a security buildup that served as a deterrent that kept the event safe. That lesson, Mr. Cohen says, informed the plans to increase capabilities, including at the intelligence office, which he was recently tapped to run.

DHS officials have stressed concerns about protecting civil liberties in discussions with dozens of research centers and government contractors, people familiar with the talks say. The department wouldn’t hire companies to monitor individuals, they say, but would seek high-level data that indicate potential targets for violence and identify foreign influence campaigns from countries such as China or Russia. The information would be anonymized, the people say.

One company involved in recent discussions with DHS, called Logically, worked on a government project last year that involved alerting U.S. election officials to online disinformation intended to dissuade Americans from voting, according to Logically Chief Executive Lyric Jain.

Mr. Jain’s company uses artificial intelligence and human analysts to monitor online content at scale, seeking trends around misinformation. The company says it can identify emerging narratives much more easily than U.S. agencies and help them respond faster.

Logically could access public channels such as those on the encrypted Telegram social network that some government officials can’t join under current rules, Mr. Jain said. His company could then share an analysis of any potential threats discerned there with U.S. agencies. Legal limits would bar private analysts from sharing personal information of American participants without a court order, he added.

Mr. Cohen said DHS wouldn’t engage with any outside group to acquire information that it is restricted from accessing. He declined to comment on the DHS intelligence office’s current restrictions on accessing social-media data, saying that doing so could allow “threat actors” to circumvent the office’s efforts.

The high-level analysis that DHS receives would bolster security, Mr. Cohen said. “You never get intelligence that tells you the specific location that’s at risk,” he said, “but you may get information that tells you that an international terrorist group or domestic extremist group seeks to target a certain faith community or certain ethnic community.”

Still, civil-liberties advocates worry that such data—even anonymized—could lead to government overreach. DHS, in intelligence reports, has previously identified as suspicious views including “opposition to gun control and immigration, anger about the 2020 election results or the use of force by police, or a belief in certain conspiracy theories,” said Rachel Levinson-Waldman, deputy director for the Brennan Center’s national-security program.

“There’s a real temptation, once the tool is in place…to say, ‘Well, we have this, why don’t we use it for these other purposes?’ ” said Ms. Levinson-Waldman.

—Michelle Hackman contributed to this article.

Write to Rachael Levy at rachael.levy@wsj.com