What was caste system like when the subcontinent was mostly Buddhist?

As the question suggests, I always wondered the reasons behind the decline of Buddha's legacy and teachings in subcontinent and its eventual takeover by Brahmanical forces.

Did caste also become an intrinsic reality within Indian Buddhist society just like what happened to Sikhism despite its total abhorrent rejection to caste?