In a large stand mixer, combine yeast, warm milk, pumpkin, butter, egg, brown sugar, and salt. Add in 2 1/2 cups of flour.
Start mixing, adding the remaining flour, half cup at a time until a nice, soft, smooth and elastic dough ball forms.
Transfer to a lightly greased bowl, cover, and let rise until double in size, about 1 hour.
Punch down the dough and roll into a 12 inch by 18 inch rectangle.
In a mixing bowl, make the filling by whipping the butter with the pumpkin puree. Mix in brown sugar, cinnamon, ginger, and nutmeg. Spread across the rolled out dough leaving 1/2 inch around the top edge. Roll up tightly lengthwise towards the top edge so you have one long roll. Use a sharp knife to cut the dough into 12 one-inch slices.
Place the slices onto a lightly greased 9x13 pan. Cover and let rise 20 to 30 minutes.
Preheat oven to 350 degrees. Bake the rolls for about 18 to 20 minutes, until just kissed with brown on top.
While the rolls are baking, make the cream cheese glaze by using a hand mixer to whip together cream cheese and butter in a bowl until light and fluffy. Whip in powdered sugar and vanilla extract. Add enough milk to achieve a drizzle-like consistency.
Frost the rolls while still warm. Serve immediately or cool and store. Stays good for 4 to 5 days.