Cool project! I looked over your code very briefly and noticed you don't tend to use destructuring very much. It was something that took me a while to get used to too, but it turns out to be very useful and makes the code much smaller. Take for example this:
(defn stats [requests-state]
"Returns a mapping of RequestsState http states states to counts"
(reduce
(fn [stats request]
(let [r @request
state (:state r)
status (:status r)
statuses (:statuses stats)]
(assoc stats
:total (inc (stats :total 0))
state (inc (stats state 0))
:statuses (assoc statuses status (inc (statuses status 0)))
:progress (if (not= state :untried)
(inc (stats :progress 0))
(stats :progress 0)))))
{:statuses {}}
(flatten (:grid @requests-state))))
Could be written like so:
(defn stats [requests-state]
"Returns a mapping of RequestsState http states states to counts"
(reduce
(fn [{statuses :statuses total :total :as stats} request]
(let [{state :state status :status} @request]
(assoc stats
:total (inc (or total 0))
state (inc (stats state 0))
:statuses (assoc statuses status (inc (statuses status 0)))
:progress (if (not= state :untried)
(inc (stats :progress 0))
(stats :progress 0)))))
{:statuses {}}
(flatten (:grid @requests-state))))