Preheat oven to 375 degrees. In a medium bowl, whisk together flour, cocoa powder, baking soda, and salt; set aside.

In the bowl of an electric mixer fitted with the paddle attachment, cream together butter and sugar until light and fluffy, about 2 minutes. Add egg, milk, and vanilla, and beat until well combined. Add reserved flour mixture; mix on low speed until combined.

Drop scoops of dough on parchment or silpat. Bake until cookies begin to spread and become firm, 10 to 12 minutes.

Remove baking sheets from oven, and place a marshmallow, cut-side down, in the center of each cookie, pressing down slightly. Return to oven, and continue baking until marshmallows begin to melt, 2 to 2 1/2 minutes. Transfer cookies to a wire rack to cool completely before frosting.

Pipe frosting on top.

Frosting:

Place confectioners' sugar in a medium bowl. Whisk in butter and cocoa powder. Add milk and vanilla, and whisk until well combined.