Settled in 1854 by Pennsylvania native Benjamin Spencer, Spencer Brook was one of the first communities in Isanti County. During the 1850s, Spencer Brook became a favorite spot for east coast Yankees seeking lumbering opportunities. The nearby Rum River provided the flowing waters needed to float logs downstream to the Twin Cities.
The first post office in the county was established in Spencer Brook in 1857 and a school in 1858. A hotel, blacksmith shop, implement store, creamery, photography shop, general store, and a grist mill soon followed. To ease crossing the Rum River, a ferry began operation in 1870. In 1874 a bridge was built to replace the ferry, but it was destroyed by ice the following spring. The brook that splits the town was dammed for the grist mill, and the resulting millpond was a popular recreation area for the residents.
Spencer Brook became known as the largest settlement of native-born Americans in the state outside of the Twin Cities. Swedish immigrants began arriving in the area in the 1870s and fields of potatoes and navy beans began to surround the township.