The European Court of Justice said Tuesday the EU’s competition authority lacked jurisdiction to review U.S. gene-sequencing company Illumina’s takeover of cancer-test maker Grail and that the legal tool it relied on to look at the case was used improperly.

Illumina has since unwound Grail, but the legal case was nonetheless watched closely because of its implications for the EU’s approach to policing deals that fall below the bloc’s and member states’ traditional thresholds for review.

The decision is expected to trigger a wider debate on how to handle deals that regulators believe could pose a threat to competition in Europe in the future, but where a company being acquired has little or no current revenue in the bloc.

The EU had sought to expand its oversight of such deals to target so-called killer acquisitions, where a company buys a much smaller player to eliminate a source of potential competition, and takeovers of small but highly innovative companies.

Business groups had complained that the EU’s approach created a fresh source of uncertainty for companies because it became more difficult to determine whether a transaction would attract scrutiny in Europe.

“It will be a cold shower for the enforcers,” Kasia Czapracka, a partner with law firm White & Case who is based in Brussels, said of the decision. “It may discourage this overreach that we’ve seen.”

The European Commission, the bloc’s antitrust enforcer, said on Tuesday that it would carefully study the court’s judgment and its implications.

“We will consider the next steps to ensure that the Commission is able to review those few cases where a deal would have an impact in Europe but does not otherwise meet the EU notification thresholds,” EU competition chief Margrethe Vestager said.

Vestager added that some member states have introduced changes in recent years that allow them to consider deals that don’t meet their usual thresholds, opening more possibilities for the types of cases the commission can look at while still complying with the court’s ruling.

Some antitrust lawyers said the court’s decision may not provide companies with much relief in the long term, in part because some national authorities have changed their approach and because the commission could still introduce new legislation to allow it to review more deals.

“The antitrust authorities of the world have become much more focused on big companies buying small ones,” said Alec Burnside, senior counsel with law firm Dechert LLP in Brussels, who has advised Illumina.

Illumina welcomed the court’s ruling, saying that it confirmed its view that the EU exceeded its authority by asserting jurisdiction over the Grail deal. Illumina also said the ruling removed the basis for a 432 million euro fine, equivalent to $477 million, that the Commission had earlier issued against the company for proceeding with the deal before the competition watchdog had determined whether to approve it.

That fine will no longer be payable, Illumina said.

U.S. antitrust officials also objected to Illumina’s takeover of Grail. Illumina said late last year that it wouldn’t pursue further appeals and would divest the business. Illumina completed its spinoff of Grail in June.

Both the U.S. Federal Trade Commission and the European Commission had raised concerns that Illumina would have an incentive to make it harder for Grail’s rivals to access Illumina’s gene-sequencing technology.

The EU’s scrutiny of the case was controversial because Grail, which is based in California, has no revenue in the bloc.

Write to Kim Mackrael at kim.mackrael@wsj.com