Brown was born in South Lee, Massachusetts, and grew up in Massachusetts and Connecticut. His was a New England merchant family. Brown entered Yale College at 16, where he was a member of Alpha Delta Phi fraternity. He earned a Bachelor of Arts degree there in 1856. Among his undergraduate classmates were Chauncey Depew, later a U.S. Senator from New York, and David Josiah Brewer, who became Brown’s colleague on the Supreme Court. Depew roomed across the hall from Brown for three years in Old North Middle Hall, and remembered “a feminine quality [about Brown] which led to his being called Henrietta, though there never was a more robust, courageous and decided man in meeting the problems of life[.]” After a yearlong tour of Europe, Brown attended a term of legal education at Yale Law School and another at Harvard Law School. Many lawyers at the time did not earn law degrees, and he did not.

Admitted to the Michigan Bar in 1860, Brown’s early law practice was in Detroit, Michigan, where he specialized in admiralty law (that is, shipping law on the Great Lakes). In addition to his private law practice, at times between 1861 and 1868 Brown served as Deputy U.S. Marshall, assistant United States Attorney for the Eastern District of Michigan, and judge of the Wayne County Circuit Court in Detroit.

In 1864, Brown married Caroline Pitts, the daughter of a wealthy Michigan lumber merchant. They had no children. He did not serve in the Union Army during the Civil War. but like many well-to-do men instead hired a substitute soldier to take his place.  Brown kept diaries from his college days until his appointment as a federal judge in 1875. Now held in the Burton Historical Collection of the Detroit Public Library, they suggest that he was both genial and ambitious, but also depressed and doubtful about himself.

The death of Brown’s father-in-law left Brown and his wife financially independent; he was willing to accept the relatively low salary of a federal judge. On 17 March 1875, Brown was nominated by President Ulysses Grant to a seat on the United States District Court for the Eastern District of Michigan left vacant by the death of John Wesley Longyear. Brown was confirmed by the United States Senate two days later and immediately received his commission.  Brown edited a collection of rulings and orders in important admiralty cases from inland waters, and later compiled a case book on admiralty law for lectures at Georgetown University. He also taught admiralty law classes at the University of Michigan Law School from 1860 to 1875, and medical jurisprudence at the Detroit Medical College from 1868 to 1871.

After lobbying by Brown and by the Michigan congressional delegation, President Benjamin Harrison appointed Brown, a Republican, to the Supreme Court of the United States on 23 December 1890, to a seat vacated by Samuel F. Miller. Brown was unanimously confirmed by the Senate on 29 December 1890, and received his commission the same day. In an autobiographical essay, Brown commented “While I had been much attached to Detroit and its people, there was much to compensate me in my new sphere of activity. If the duties of the new office were not so congenial to my taste as those of district judge, it was a position of far more dignity, was better paid and was infinitely more gratifying to one’s ambition.”

As a jurist, Brown was generally against government intervention in business, and concurred with the majority opinion in Lochner v. New York striking down a limitation on maximum working hours. He did, however, support the federal income tax in Pollock v. Farmers’ Loan & Trust Co. (1895), and wrote for the Court in Holden v. Hardy, upholding a Utah law restricting male miners to an eight-hour day.

Brown is best known for the 1896 decision in Plessy v. Ferguson, in which he wrote the majority opinion upholding the principle and legitimacy of “separate but equal” facilities for American blacks and whites. Plessy, which provided legal support for the system of Jim Crow Laws, was overruled by the Court in Brown v. Board of Education in 1954. When issued, Plessy attracted relatively little attention, but in the late twentieth century it came to be vilified, and Brown along with it.

Justice Brown authored the Court’s 1901 opinion in DeLima v. Bidwell, one of the Insular Cases, considering the status of territories acquired by the U.S. in the Spanish-American War of 1898.

In 1891, he paid $25,000 to the Riggs family for land at 1720 16th Street, NW, in Washington, D.C., hired architect William Henry Miller, and built a five-story, 18-room mansion for $40,000. He would live in this house, later known as the Toutorsky Mansion, until his death. Ironically—in light of Brown’s racial attitudes—the house is now the embassy of the Republic of the Congo. Brown’s wife Caroline died in 1901. Three years later, Brown married a close friend of hers, the widow Josephine E. Tyler, who survived him.  Near the end of his years on the Court he largely lost his eyesight. Brown retired from the Court 28 May 1906 at the age of 70.  Brown died of heart disease on 4 September 1913, at a hotel in Bronxville, New York. He is buried next to his first wife in Elmwood Cemetery in Detroit.