The Eli Whitney barn, built in 1816, was the centerpiece of the Whitney farm. It was the focus of agricultural activities which supplied the boarding house kitchen and provided shelter for the draft animals used at the Armory. Whitney's friend Silliman considered it "a model of convenience and even taste and beauty," and further described the unique features of its design:
It is perfectly characteristic of Mr. Whitney that his attention was directed to the mangers for his cattle, and to their fastenings. The latter are so contrived, by means of a small weight at the end of the halter, that an animal could always move his head with perfect facility, but could not draw out the rope so as to become entangled in it, nor could he easily waste his hay. The fastenings of the doors, as well as all the other appendages and accommodations are equally ingenious.
That Whitney took an active role in the design and construction of the non-Armory buildings on his site is thus well established. The need for housing to accommodate his workforce was obvious given the distance of the Armory from any other suitable lodgings. The farm was necessary to provide food for the unmarried workers and probably supplemented the gardens of the married workers living on Armory Street.