Mr Abbey, who has written a book about his experiences, Think Like a White Man, says the lack of black people at the top in business is a big part of the problem, and like others he is calling for change.
On his first banking job, he was mistaken for a security guard.
It was just one of many examples of racism he experienced as a black man working in the City and later for big media companies, including shocking pay disparities and few opportunities for promotion.
It says that more than a third of the boards of the biggest listed UK businesses are still all-white, with that rising to two thirds in smaller public companies.
One of the UK's biggest investment firms L&G says it will use its vote to put pressure on bosses who don't make their top teams more diverse by 2022, putting the people responsible for board appointments in the firing line.
"You need people on the board who actually reflect your customers and society, but a lot of boards look like nothing has changed since the 1970s," says Mr Abbey.
"You think, there are almost two million black people in the country, surely one of those people are good enough to be on your board."