Legend has it that Dhanvantari was born when the devas and asuras churned the ocean. He is considered to be an incarnation of Lord Vishnu and has four hands - in one he holds medicinal herbs and in one the text of Ayurveda. He was the first Baidya. Another legend about his birth is that when the sage Galava was on a pilgrimage and dying of thirst, he accepted water from a Vaidya girl named Virbhadra. He blessed her and said she would have a son. But as she was unmarried, Galava chanted a Vedic mantra and a boy arose out of a wisp of grass and it was this child who was Dhanvantari. He is also known as Amritacharya or Ambastha. He was a Baidya because he owed his birth to a Vedic hymn and he was an Ambastha because he had no father and was brought up by the mother and her family. So Baidyas are also called Ambasthas. But Baidyas say that Ambastha is the name of a place on the banks of the river Indus from where one branch of Baidyas went to South India and another to Bengal (Goud).

There is still another legend, which describes Baidyas as begotten of a Brahmin woman by the Ashvins, the gods of healing. Their union being `pratiloma` in which the father is inferior in caste to the mother the offspring was not Brahmin. Even the gods were considered inferior to the Brahmins. Originally the practice of medicine all over India was mostly in the hands of Brahmins.

Medicine was the original profession of the Baidyas, but from the time of the Sena kings, they began to adopt other pursuits as well. A Baidya by caste need not only be a physician.

The Sena kings were probably Baidyas. The evidence of inscriptions shows that a dynasty of Baidya kings ruled over at least a part of Bengal from 1010 AD to 1200 AD. The most famous of these kings is Ballal Sena. He separated the Baidyas into divisions: the first group was allowed to wear the sacred thread and fifteen days observance of mourning while the other group has an option to adopt the thread and to observe the mourning for a month. He is also said to have made three classes -Rarhi, Barendra and Bangaja. This was done by him according to the place of residence and introduced three hypergamous divisions - Kulin, Bangsaj and Maulik.

He is also supposed to have instituted 3 other rigid divisions of purity of lineage-the Siddha, Sadhya, and Kashta. The nomenclature is interesting because, according to a legend, Amritacharya or Dhanvantari married the Ashvin`s three daughters-Siddha Vidya, Sadhya Vidya and Kashta Vidya. Even in recent times, the Baidyas of these `sthanas` consider themselves higher than others. Those who went farther east to places like Tripura, Sylhet and Chittagong were looked down upon still further. They intermarried with the Kayasthas probably because in those distant parts they could not always get a suitable match within their own clan.

Centuries later, Raja Raj Ballav Sena was not only the undisputed leader of the Baidya community but a great Indian whose name will always find a place in history. He was appointed collector general of the province of Dacca and given the title of Raja by the Nawab of Murshidabad. Shah Alam, the Mughal Emperor of Delhi, made him a maharaja with the title Rai Raiyan Salar Jung Bahadur and presented him with a sword of honour. He played an important role in political history and court intrigues during a chequered period. When Mir Jaffar succeeded Siraj-ud-daula with the help of the British, he made Raj Ballav his minister. He tried to get rid of the British but Mir Qasim`s men drowned him and his son, Krishnadas Sena, in the Ganga when he was the Subedar of Monghyr.

He persuaded some Brahmins to invest his son with the sacred thread after which many Baidyas wore this badge of distinction. For several generations, the leadership of the Baidyas has been vested in his family, which had its seat at Rajnagar on the south bank of the River Padma. The Baidyas retained among their group not only rajas and maharajas and powerful zamindars (landlords) but also scholars of great distinction such as the five gems at the court of Lakshman Sena, two of whom were Jayadeva, the famous composer of the Gita Govinda and Dhoyee Kaviraj.