Thousands of people had packed into the Puttingal Devi temple and the narrow lanes around it. The fireworks display began at midnight and went on for hours.
One of the crackers then fell on a shed where the fireworks were stored, setting off an explosion and the fire, local television reported. The temple is located in Kollam district, about 70 km from the state capital Thiruvananthapuram.
Kerala’s Chief Minister Oomen Chandy said 79 people were dead, while many were trapped by flames in the temple building and 245 were injured.
Television images showed people, some clutching children with burn injuries, being taken to hospitals.
Prime Minister Narendra Modi said he would be flying to the temple site later on Sunday.
“The fire at the temple in Kollam is heart-rending and shocking beyond words,” he said in a Twitter post. “My thoughts are with families of the deceased & prayers with the injured.”
Kerala is in the midst of an election for a new assembly as politicians wrangled over the cause of the fire.
Modi’s Bharatiya Janata Party trails its rivals – the Congress party and the communists. It has focused on building up grassroots strength in the state for decades.
Chandy said the Kollam district administration had denied the temple authorities permission to carry out the fireworks display, and that the government will investigate how they went ahead.
“There was no permission to even store the fireworks,” he said.
Kerala is studded with temples, managed by rich and powerful trusts that often flout local regulations. Each year temples carry out fireworks displays, often competing to stage the most spectacular ones.
Kollam district magistrate A. Shainamol said people living in the neighbourhood of the temple had complained about the danger of these fireworks in the past.
The explosion from the fireworks was so strong that some parts of the temple roof caved in. Local media showed bulldozers trying to clear the area of the debris.