Graham Wrighting admits he fears Spencer will fold when he walks away from his role as chairman.
The 72-year-old has been at the helm for an incredible 33 years, but is now keen to take a step back.
However, with an ever-decreasing committee and little interest from outside, Wrighting has genuine worries for the future of the UCL Premier Division club.
He said: “We’re running out of people!
“You need people to mow the pitch, look after the bar, look after the football side of things, but I’m 72 years old now and I’m not going to live for ever.
“You used to get players going on the committee when they stopped playing, but now they just disappear out of sight.
“If neither myself or John Sampson (the groundsman and committee member, who is stepping aside after 45 years at the club) do anything, the club will fold.
“You don’t want to see that happen, but what can you do?
“I think we’ll struggle because I can’t see anyone coming in.
“It’s been known I’m wanting to step aside for a month now, but we’ve had no interest from anyone.
“I don’t want to see the club go and I’ll keep doing my bit.”
Wrighting is happy to look after the football part of the club, but he does not want the burden of the social side, too.
One possibility is that Spencer will link up with Sunday side the Falcons, who could take over the running of social events.
“It’s in the pipeline,” Wrighting said.
“It would be nice for me to go to a game and not have to do anything and I’d also like to have the option not to go.
“I still want to stay involved, but I’d rather people take over certain things.”
Long-serving Sampson and programme manager Andy Goldsmith, who is going to take a back seat after 32 years at the club, will also be a huge loss to the club.
And Wrighting said: “John’s done everything at the club.
“He’s been here for 45 years and I always said that when he stepped down, I would, too.
“Me, John and Andy have all served our prison sentences!”
Anyone wanting to take up a role as chairman, groundsman or committee member can register an interest with Wrighting on 07711465485.