More than half (56%) of people say they would cycle more if roads had fewer faults such as potholes, according to research commissioned by charity Cycling UK.
Some 22 cyclists were killed and 368 seriously injured where a poor or defective road surface was a factor between 2007 and 2016, Department for Transport figures show.
One in five local roads in England and Wales is in a poor condition as councils face a huge funding deficit to tackle potholes, a recent report by the Asphalt Industry Alliance warned.
Father-of-two Simon Moss, 44, lost four teeth and fractured his spine after crashing into a pothole that was reportedly 9in deep while cycling in Buckinghamshire last month.
Cycling UK's head of campaigns Duncan Dollimore said: "Cycling is still a minority activity in the UK with only 2% of all journeys made by bike.
"Those who do cycle put up with the potholes and dangerous traffic conditions daily and still continue. However, it's not always pleasant and it's no surprise most people do not consider cycling for their short everyday journeys."
The charity's poll of 2,024 adults, conducted by YouGov, found that poor road conditions was the joint second most common reason for avoiding travel by bike, alongside drivers overtaking too closely.
Having to share the road with lorries and other large vehicles was ranked number one with 57% of respondents put off by it.
Other concerns include threatening behaviour from drivers (43%), car doors being opened in front of them (40%) and speeding motorists (37%).
The research was released to coincide with Cycling UK's submission to the Government's Cycling and Walking Investment Strategy safety review.
It wants "fundamental changes" to areas such as the Highway Code, road design and vehicle safety to give people more confidence to cycle.
Mr Dollimore added: "The Government wants more and safer cycling, but as Cycling UK's research shows, people who don't currently cycle need change if they're going to choose to cycle short distances rather than drive."
RAC spokesman Simon Williams said: "Potholes are bad enough for those in vehicles, but for anyone on two wheels they are an accident waiting to happen.
"Councils should be doing all they can to make sure they identify and fix potholes so lives aren't ruined or lost when cyclists unwittingly ride into them."
Martin Tett, transport spokesman for the Local Government Association, which represents more than 350 councils in England and Wales, said: "Councils are fixing a pothole every 21 seconds and keeping roads safe for cyclists is one of the most important jobs we do.
"However, only long-term, consistent and fairer government investment in local road maintenance can allow councils to embark on the widespread improvement of our roads that is desperately needed."
A Department for Transport spokeswoman said: "We have some of the safest roads in the world, but we will do everything we can to make our roads safer for everyone, including cyclists.
"That's why we are investing a record £23 billion on our roads to increase capacity and improve journeys.
"This includes providing local highway authorities in England with over £6 billion to help improve roads, including a record £296 million to help fix potholes and stop them forming.
"While it is for councils to identify where repairs should be undertaken, we are also looking at how innovative technology can help them keep their roads in the best condition and save money."