Exit polls suggest that Danish voters have voted against adopting EU rules strengthening cross-border policing.
Voting finished at 19:00 GMT. An exit poll released soon afterwards by the TV2 news station showed 52.8% of Danes voted No.
Another exit poll, from the DR station, showed 53.3% voted against the proposals.
Denmark’s centre-right government wants to abandon some Danish opt-outs from EU home affairs legislation.
The vote comes three weeks after the deadly attacks in Paris.
The government, backed by the opposition, had said a No vote would mean losing membership of the Europol crime agency.
But the anti-immigration Danish People’s Party (DPP), which props up Liberal Prime Minister Lars Lokke Rasmussen’s administration in parliament, had called on voters to reject the proposals, to avoid giving away further sovereignty to Brussels.
The referendum is seen as a test of whether the Danes will accept or reject greater integration within a 28-member bloc that is being tested more than ever by the migrant crisis, the BBC’s Gavin Lee reports from Copenhagen.
Although a Yes vote would not affect Denmark’s opt-out on immigration, the DPP argues that it could eventually lead to immigration policies being dictated by the EU.
Unlike Denmark, the UK and Ireland have opt-ins on justice and home affairs legislation, which enable them to choose whether to accept or reject legislation on a case-by-case basis.