The overall quality of experience of GP practices was most strongly associated with factors including how well doctors listened to patients and explained treatments, according to a study published in the British Journal of General Practice.
The authors warned that while telephone and video consultations may improve GP access, the changes could have the "unintended consequence" of affecting the quality of the doctor-patient relationship.
They also said improving opening times and making it easier to get an appointment would not necessarily result in "large improvements" in patient satisfaction.
The researchers, using data from the General Practice Patient Survey, analysed responses from 2.9 million patients from more than 8,000 UK practices between 2011 and 2014.
Patients' overall experience of GP surgeries was most strongly associated with the interpersonal quality of care provided by doctors, they found.
This included how much time GPs gave to patients and how much they involved them in decisions, as well as how well they listened and explained tests.
A patient's experience of making appointments and satisfaction with opening hours was only "modestly associated" with their overall experience, according to the research.
The authors, from Imperial College London and the London School of Hygiene and Tropical Medicine, said: "Policymakers should reflect on this finding and consider the contexts of GPs' work that affect interactions with patients.
"For example, a large workload could affect whether GPs can give each patient enough time at consultation."
They added: "Some interventions being promoted to improve access to general practice, such as telephone and video consultations, change the GP-patient interaction substantially - an unintended consequence could be reduced interpersonal quality of care."
The results of the latest British Social Attitudes survey revealed only 65% of people in England, Scotland and Wales were satisfied with GP services - the lowest level since records began in 1983.
NHS England committed to ensuring everyone has access to weekend and evening routine GP appointments by 2020.
However, the authors said: "It is suggested that policymakers should not assume that recent national policies focused on access to general practice will translate into large improvements in patients' overall experiences, even if they do actually improve access."