Hundreds of protesters gathered outside a hotel in Epping on Sunday for the fifth time to protest at the premises being used to house asylum seekers, as protests spread to other hotels over the weekend.

A large police presence containing officers from multiple forces restricted contact between anti- and pro-immigrant protesters, with Essex police saying restrictions were necessary after what it described as repeated serious disruption, violence and harm to the community since the first demonstration took place on 13 July.

Two men have been charged with public order offences after a protest of about 400 anti-immigration and 250 counter-protesters outside a hotel in Diss on Saturday, Norfolk constabulary said. There was a further protest outside a hotel in Canary Wharf, London, on Sunday, with the number of protesters appearing to be in the low hundreds.

The demonstration in Epping, Essex on Sunday – which saw about 300-500 anti-immigrant protesters gather behind metal barriers outside the Bell hotel – was the latest in a series of protests sparked after an asylum seeker was charged with sexual assault for allegedly attempting to kiss a 14-year-old girl. Hadush Kebatu, 41, from Ethiopia, has denied the offences and is in custody.

Protesters wore T-shirts and held up signs with the slogan “Protect our kids”, while others waved England flags. Other flags seen included one for Reform UK, and a white flag with a red cross on a purple square, as seen in America at anti-abortion demonstrations.

Counter-protesters held banners including “Don’t let the far right divide us with their hatred and violence”, and “Care for refugees”. They chanted “Refugees are welcome here” and “Nazi scum off our streets”.

Police said three people were arrested during the peaceful protest, two from the anti-hotel protest group and the other from the counter-protest group. “I want to thank those who attended for the peaceful nature of both protests,” said Ch Supt Simon Anslow of Essex police. “I am pleased that today has passed off without incident and I am grateful to our colleagues from other forces for their support.”

In a letter sent to the Guardian, asylum-seekers said “harmful stereotypes” about refugees did not reflect the truth.

“There are some refugees who do not behave respectfully or who do not follow the rules of the host society. But those individuals do not represent all of us,” they said. “As with any group of people, there are both good and bad – and it is unfair to judge the majority by the actions of a few.”

The letter mentioned fleeing persecution and violence. “We refugees are not here to take advantage of the system. We are here to rebuild our lives, to work, and to contribute,” they wrote, adding: “This letter is not a plea for sympathy, but a call for understanding and fairness.”

Outside the Bell hotel, one local woman, who did not want to be named, said local people had complained about an increase in antisocial incidents since it began housing asylum seekers but felt ignored and unfairly labelled as “far-right”.

“I’m not saying everyone in any of these hotels is up to no good. I’m not going to judge everyone, but there is no vetting,” she said. “We won’t stop until they start listening and shut this hotel down.”

Activists from far-right groups including Homeland, Patriotic Alternative and the neo-Nazi White Vanguard movement have been present at previous protests. On Sunday Kai Stephens, the Norfolk branch organiser for Homeland, held a sign which said: “Put local people first.”

Stephens said: “Unfortunately, there has to be a certain point where we turn around and say, the British people should be put first, the indigenous British people.”

Supporters of the far-right activist Tommy Robinson were also present. Robinson, 42, whose real name is Stephen Yaxley-Lennon, had said he would attend before changing his mind and saying it would not be helpful to protesters.

Wendell Daniel, a former Labour councillor who is now a film-maker for Robinson’s Urban Scoop video platform, asked one man if he understood why Robinson had not attended. He responded: “We’re with Tommy all the way.”

Other local protesters said that far-right agitators were not welcome. “It’s 100% unhelpful, because it just gives them a message which is not what we’re trying to achieve here,” said one man, who did not want to be named.

Stand Up to Racism, the group that organised the counter-protest, estimated about 700 people had gone to Epping. Lewis Nielsen, an officer at Stand Up to Racism, said Nigel Farage’s Reform UK had emboldened the far right.

“It’s a really dangerous situation at the moment because you haven’t just got the protest here, you’ve got other protests coming up around the country,” Nielsen said. “We stopped the riots last August with these kinds of mobilisations, and that’s why we’re pleased the one today has been successful.”

Joshua Bailey, who said he grew up in Epping, said increasing anti-immigration sentiment had made his non-white friends feel vulnerable and threatened. “It’s very important that we have a positive stance towards refugees, who are people fleeing genuine tragedy and disaster,” he said.

He added that he did not agree with chants that labelled protesters as fascists or Nazis. “There is room for nuance,” he said. “I’d like to be able to sit down in a pub with someone who had opposing views and be able to speak about it.”