Close to 50 per cent of people on zero-hours contracts, and two in five people on temporary contracts wrongly believe they cannot claim a wage while they take a break, the charity has revealed in a recent study.
Central and East Northamptonshire Citizens Advice also says that some employers are even deliberately "flouting" and "exploiting workers’ confusion".
The charity is now calling on the next government to ensure workers are aware of, and are able to take, the paid holiday they are entitled to.
Chief executive Martin Lord, said: " Half of people on zero-hours contracts, and many on temporary contracts, think they are not entitled to paid holiday.
“We already encounter significant demand from clients with problems related to this.
"But with Northamptonshire being very well represented by the warehousing and distribution sector and other sectors known to utilise these kinds of contract as well as a significant migrant worker population, we think what we see through our advice services can only be the tip of the iceberg."
In the last financial year almost 185,000 people got help from Citizens Advice on employment issues - with 10,000 cases specifically about paid holiday.
Over the same period, the Citizens Advice web page on paid holiday had 260,000 visitors.
One man the charity helped worked in a care home for over five years, working 48 hours a week. His employer had previously told him that night workers were not entitled to paid holiday, which he had not questioned until recently.
When he visited Citizens Advice, advisors worked out he had missed out on paid holiday of £8,900.
The charity also claims one woman who worked in the sales sector was told she could only take holiday if she met her sales targets, which is unlawful.
In the UK, between a third and half of employers use fixed-term or temporary workers and 57 per cent of employers use variable hours or shift work.
Citizens Advice is calling on the next government to combine the enforcement of employment rights into a "fair work authority" that can tackle the employers that break the rules.
It also wants to place a £50 cap on employment tribunal fees and define what "self-employed" means in law to prevent workers being exploited by their bosses.
“With more than half of employers having staff working shifts or variable hours, action needs to be taken now to protect workers rights," Mr Lord added.
"There’s been welcome attention from political parties on issues surrounding rights at work, and we hope that the next Government takes steps to make people’s jobs and income more secure."