McCoy was born in Colchester, Ontario, Canada, to former slaves (George and Emillia McCoy), who had fled the USA. McCoy was educated as a mechanical engineer in Edinburgh, Scotland; he then moved to Detroit, Michigan, USA. Despite his education (and due to racism - he was of African descent), he could only get a job as a fireman and oilman on a steam-engine train, shoveling coal into a train's engine and periodically lubricating the engine.

McCoy's first invention (1872) was a revolutionary way of lubricating steam engines without having to shut them down - this automatic lubricator saved an enormous amount of time and effort in transportation and in industrial production. McCoy eventually had a total of 57 patents, and was known throughout the world for his inventions. In 1920, McCoy opened his own company, the Elijah McCoy Manufacturing Company.