The band was formed in 2001 by Toma Feterman, who has ancestors of Polish, Rumanian and Jewish Russian origins. A self taught multi-instrumentalist, Toma sings as well as playing the banjo, the guitar and the  trumpet.
After initially performing music that covers a wide range from punk to French “chanson”, he decides to gather a group to perform music from Eastern Europe - a way of returning to his roots. At the beginning, the musicians were friends that played traditional songs. He meets Olivier Llugany in January 2001. Llugs started playing the fiscorn (a traditional bass bugle from Catalonia) at a young age, in the Sardane orchestras of his region (coblas). As a result, this unusual instrument has been introduced into the project – it has certain similarities with the tubas of the brass bands from the Balkans. It is thanks to Llugs that the group has that particular brass sound. Between 2001 and 2003, the band becomes popular in Paris, acquiring a reputation as a celebratory gypsy-klezmer ensemble. They can be found on all the stages of the Paris musical underground scene.

During 2002, La Caravane Passe creates, together with Môh Aroussi, « le Vrai-Faux Mariage » (the Real-Fake Wedding), a kind of gypsy rock cabaret performance.
In 2003 Ben Body, a great friend of Toma Feterman, becomes bass and double bass player for the band.
2004 is the great turning point in the group’s career : that year, Pat Gigon and Cyril « Zinzin » Moret join La Caravane Passe : the set-up is complete.
As they meet up and listen to gypsy bands across France, Toma and Llugs observe that this music has a great impact on the French public, but remains closeted in a finally rather cultural, conventional context. This is when the idea hits them to “translate” this music, to move it without insulting it, to deviate it towards the French rock idiom. Toma’s compositions are rather alternative rock songs that have taken the colours of gypsy music in the wider sense (from Central Europe to Spain and Northern Africa). The music is full of energy and very festive, wandering from Balkan tango to hip-hop, from reggae to tarantella.

Since 2004, the band have been on stage with some of the most famous gypsy bands: Fanfare Ciocarlia, Taraf de Haïdouk, many collaborations with Rona Hartner, Kocˇani Orkestar, O’djila, Boban Markovic, Bratsch, Slonovski Bal, Goran Bregovic … but also with French rock bands like Sergent Garcia,  Java, Mon Côté Punk, the Washington Dead Cats, El Gafla, Le Grand Orchestre du Splendide… For two years in a row, the band filled the well known Paris venue Cabaret Sauvage with the “Real Fake Wedding” once a month, and also played in Switzerland, Belgium, Germany, as well as in Serbia and Macedonia; and they also played in famous Paris venues like the Elysée Montmartre, Bataclan or New Morning.