The find prompted police to block off an area near Jones Bay at Pyrmont as military personnel seized the device.

Dive instructor Tony Strazzari said he'd plunged into the water for a routine dive with friend Paul Szerenga when he spotted the bomb's cylindrical shape and dart-like end nestled on the harbour floor.

"We do a lot of diving in old war areas up and down the coast, mostly looking for old bottles, black glass, embossed bottles, glassware," Mr Strazzari said.

"We were just swimming there and poking around the silty bottom.

“I reached in and felt something heavy and pulled it out and it was a bomb."

Worried that the device might be disturbed by a boat anchor if it stayed beneath the water, Mr Strazzari tucked the 40-centimetre bomb into his diving gear.

His diving mate wasn't so unfazed: "When he handed it to me under the water I said, 'Holy shit'," Mr Szerenga said.

The pair alerted authorities when they got out of the water before they "baby-sat" the device until police arrived.

"We sat there and coddled it up so no-one got worried. One man did ask what it was and when I told him it was a bomb he just shrugged and walked off.

"It wasn't until the cops showed up they realised it was a bomb."

Police set up a 100-metre exclusion zone in the area, near Pirrama Park, as they called in military personnel to inspect the device. They later reduced the shut-off area to 20 metres.

A NSW Police spokeswoman said the bomb was "unable to be detonated" and not deemed a safety risk.

It was seized by military personnel about 2pm.

Mr Strazzari said friends had guessed the bomb was a three-inch mortar shell.

"I've found bullets before and I found a rifle once, but never a live artillery shell. This was a first for me," he said.

The Defence Force has been contacted for comment.