#+title: FimFinder Source
#+author: Robert McIntyre
#+email: rlm@mit.edu 
#+description: advanced search for fimfiction.net
#+keywords: my little pony search linear programming web design
#+setupfile: ../../aurellem/org/setup.org
#+include: ../../aurellem/org/level-0.org
#+babel: :mkdirp yes :noweb yes

This is the source for [[./search.html][fimfinder.net]], a search engine for
[[http://fimfiction.net][fimfiction.net]]. 

* Organization

Want a website that enables easy top-X lists and statistics
for the stories on [[fimfiction.net]].

- working-name:  [[./search.html][fimfinder.net]]

** Features

 - Lists of the top pony stories, where "top" is determined
   by multiple different metrics.
 - Use minimal bandwidth from fimfiction.net
 - Precalculate all results; this website is just a
   convenient view on info that has already been calculated
   (to ensure low bandwidth on my server and stability of
   the website.)
 - Literate Programming -- every aspect of the site from
   html to backend should be open source and available
   through the website.
 - Link back to fimfiction.net for the actual stories; the
   site can not store any of the stories themselves, just
   the metadata and statistics of the stories.

** Implementation
 - Two pages : a search page and the literate programming page.
 - Backend clojure, frontend HTML/jquery
 - There's only at most ~300,000 stories, so just use
   individual files to store the statistics of each story
   rather than a database (~500MB tops).
 - Use ajax to only download the particular list that the
   user wants.
 - Use tiered lists to reduce bandwidth (such as 16, 64 and
   512 for each statistic).
 - Store list output as list formatted html in separate files
   which is included via jquery.

* Search Page
This is the main page of the site and the one that will
appear when the user goes to   [[./search.html][fimfinder.net]]. It has a
brief description of the purpose of the site and how to use
the search functions.

The search section should already be displaying some results
using the "best" algorithm, and provide options to change
the algorithm and the number of stories which are displayed,
up to a reasonable minimum (maybe 500?)

** Useful Utility Functions

Would be nice to get exactly six such functions, one for
each pony.

 - straight up total likes (accumulated popularity) {rarity}
 - (- likes dislikes) per word (density of goodness) (use
   word count instead of chapter count to compensate for the
   fact that some stories have much longer chapters than
   other stories.) {applejack}
 - ratings over total time the story has existed (staying
   power) {fluttershy}
 - ratings per view (most remarkable) {rainbow dash}
 - total number of comments (most notable) {pinky pie}
 - linear ML utility function over combinations of the other
   search utility functions (overall goodness) {twilight
   sparkle}


* Clojure Backend

The clojure backend must do two things:
 - get story metadata from fimfiction.net
 - generate lists of the best stories by various criteria.

There are four independent dimensions by which a user can
select search results:

 - Search Algorithm
 - Mature?
 - Complete?
 - Number of Results

Both the search results and story metadata are small enough
that the information can just be stored in files instead of
a database without any real loss in performance. There are
currently only around 40,000 stories in fimfiction.net, so
one file per story is quite reasonable. Likewise, 6 search
algorithms and three choices for each other option gives 162
different possible results which will comfortably fit in one
file for each result.

Thus, the clojure backend is all about maintaining a database
of files, each of which represents a story, and creating a
database of search results, one for each combination of
story-searching options.

** Story Database

These functions define the file-based database for story
metadata. Each story goes in a file as a json string and is
read as a clojure map. The database can be updated with
minimal use of fimfiction's bandwidth by first reading all
the stories into memory, then only updating those stories
that are valid.

#+name: pony-clj-db
#+begin_src clojure
(in-ns 'org.aurellem.pony)

(def database-root 
  (File. "/home/r/proj/pony-stories/backend/db"))

(defn pony-header-net
  ([count pony-num]
     (if (zero? count)
       nil
       (try 
         (slurp
          (URL.
           (str
            "http://www.fimfiction.net/api/story.php?story="
            pony-num)))
         (catch Exception ex
           (error ex "bad download, count = " count)
           (Thread/sleep 3000)
           (pony-header-net (dec count) pony-num)))))
  ([pony-num]
     (log :info (str "Download " pony-num))
     (pony-header-net 4 pony-num)))

(defn pony-header-file [pony-num]
  (File. database-root (str pony-num ".h")))

(defn write-pony-header! [pony-num]
  (spit (pony-header-file pony-num)
        (pony-header-net pony-num)))

(defn pony-header [pony-num]
  (clojure.data.json/read-json
   (let [target (pony-header-file pony-num)]
     (if (.exists target)
       (slurp target)
       (do (write-pony-header! pony-num)
           (slurp target))))))

(def stale-days 5)

(defn new-pony-header! [pony-num]
  (let [days-old
        (.getStandardDays
         (Duration.
          (Instant. (.lastModified
                     (pony-header-file pony-num)))
          (Instant.)))]
    (if (> days-old stale-days)
      (write-pony-header! pony-num)
      (log :info (str pony-num " is " days-old
                      " days old; not downloading")))))

(alter-var-root #'pony-header memoize)

(defn valid? [header]
  (not= header (pony-header 0)))

(def dead-margin 50)

(defn clear-header! [pony-num]
  (log :info (str "Delete " pony-num))
  (.delete (pony-header-file pony-num)))

(defn pony-header-count []
  (dec (count (file-seq database-root))))

(defn clear-dead-margin! []
  (let [frontier-end (pony-header-count)
        frontier-start
        (max 0 (- frontier-end dead-margin 2))]
    (log :info "clearing dead margin.")
    (dorun
     (map clear-header!
          (range frontier-start frontier-end)))))

(defn download-new-stories!
  []
  (log :info "Downloading new stories.")
  (clear-dead-margin!)
  (loop [current-point (pony-header-count)]
    (log :info
         (str
          current-point "-"
          (dec (+ dead-margin current-point))))
    (let [headers 
          (map pony-header
               (range current-point
                      (+ dead-margin current-point)))
          num-valid (count (filter valid? headers))]
      (log :info (str num-valid " valid"))
      (if (not= 0 num-valid)
        (recur (+ dead-margin current-point))
        (log :info "done.")))))

(defn update-existing-stories!
  []
  (dorun
   (map new-pony-header!
        (filter (comp valid? pony-header)
                (range (pony-header-count))))))

(defn update-header-db! []
  (update-existing-stories!)
  (download-new-stories!))

#+end_src

** HTML Formatting

These functions define the mapping between the json-encoded
story headers and HTML.  I use [[https://github.com/weavejester/hiccup/][hiccup]] to generate the HTML
which will be loaded into the results section via
javascript. =pony-header->li= transforms a single json
header into HTML. The HTML is structured as a list of
important information about the story, with the Title and
Author first, followed by lists of important statistics
about the story. The entire story element is a link to the
actual story on fimfiction.net. To transform multiple
headers into HTML, transform each individual header into
HTML and then assemble them all together into an HTML list
using =pony-headers->html=.

#+name: pony-clj-html
#+begin_src clojure
(defn date->str [date]
  (.format
   (java.text.DateFormat/getDateInstance)
   date))

(def results-root
  (File. "/home/r/proj/pony-stories/html/results"))

(def default-story
  "A story which will not win by any reasonable metric."
  {:story
   {:status "Incomplete",
    :full_image "", 
    :words 1,
    :chapters
    [{:id 0, :title "No Chapter Title", :words 1,
      :views 0, :link "",
      :date_modified 0}],
    :author {:id "0", :name "Nobody"},
    :image "",
    :likes 0,
    :total_views 0,
    :title "No Title",
    :dislikes 0,
    :views 0,
    :url "",
    :categories
    {:Romance false, :Dark false,
     (keyword "Slice of Life") false,
     :Sad false, :Human false, :Random false,
     (keyword "Alternate Universe") false, :Comedy false,
     :Tragedy true, :Crossover false, :Adventure false},
    :date_modified 0,
    :chapter_count 1,
    :short_description "No Short Description",
    :content_rating 0,
    :comments 0,
    :content_rating_text "Everyone",
    :id 0, :description "No Description"}})

(defn sanitize [header]
  {:story (merge (:story default-story) (:story header))})

(defn pony-header->li
  [header index]
  (let [story  (:story (sanitize header))
        
        categories
        (sort
         (map (comp name first)
              (filter (comp (partial = true) second)
                      (:categories story))))

        title (:title story)
        author (:name (:author story))
        url (:url story)

        description (:short_description story)

        status (h (:status story))
        num-chapters (h (:chapter_count story))
        num-words    (h (:words story))
        content (h (:content_rating_text story))
        id      (h (:id story))        

        likes (h (:likes story))
        dislikes (h (:dislikes story))
        total-views (h (:total_views story))
        comments (h (:comments story))
        last-modified
        (h
         (date->str
          (java.util.Date.
           (long (* 1000
                    (:date_modified story))))))]
    (html
     [:li {:class "pony-list-item"}
      [:a {:class "pony-link" :href url}
       [:div {:class "pony-list-item"}
         
        [:div {:class "pony-header"}
         [:em {:class "pony-title"}  title " "]
         [:em {:class "pony-by"} " by "]
         [:em {:class "pony-author"} author]
         [:div {:class "pony-description"} description]]
        
        [:div {:class "pony-info"} 

         [:div {:class "pony-summary"}
          [:ul {:class "pony-summary"}
           [:li {:class "pony-status"} status]
           [:li {:class "pony-content"} content]
           [:li {:class "pony-chapters"}
            "Chapters: " num-chapters]
           [:li {:class "pony-words"}
            "Words: " num-words]
           [:li {:class "pony-id"}
            "Modified: " last-modified]]]
         
         [:div {:class "pony-stats"}
          [:ul {:class "pony-stats"}
           (map (fn [desc value]
                  [:li
                   [:div {:class "pony-stats-container"}
                    [:div {:class "pony-stats-desc"} desc]
                    [:div {:class "pony-stats-value"} value]
                    ]])
                ["Views" "Likes" "Dislikes" "Comments" "ID"]
                [total-views likes dislikes comments id])]]

         [:div {:class "pony-categories"}
          (if (empty? categories) "&nbsp"
              [:ul {:class "pony-categories"}
               (map #(vector :li %) categories)])]
         [:div {:class "pony-index"}
          [:div {:class "inner-index"} index]]]]]])))

(defn pony-headers->html [headers]
  (html [:ol {:class "pony-list"}
         (map pony-header->li headers
              (range 1 (inc (count headers))))]))
#+end_src

** Utility Functions

Here is where the stories are sorted according to the six
different utility functions and the static HTML result files
are created.


There are 5 basic utility functions, =overall-ratings=,
=total-comments=, =ratings-per-view=, =ratings-per-word=,
and =likes-per-total-time=, which each compute a simple
numerical value ranking the story in relation to other
stories. For example, =ratings-per-word= returns:

$$\frac{\text{likes} - \text{dislikes}}{\text{total number of words}}$$

for each story.

#+name: pony-clj-utility
#+begin_src clojure
(defn overall-ratings
  [header]
  (:likes (:story header)))

(defn total-comments
  [header]
  (:comments (:story header)))

(defn adjusted-ratings
  [header]
  (- (:likes (:story header))
     (:dislikes (:story header))))

(defn ratings-per-view
  [header]
  (let [views (:total_views (:story header))
        adjusted-views (if (< views 100) 1e9 views)]
    ;; penalize stories with very few views.
    (/ (adjusted-ratings header) adjusted-views)))

(defn ratings-per-word
  [header]
  (let [words (:words (:story header))
        adjusted-words (if (< words 250) 1e9 words)]
    ;; penalize stories with fewer than 250 words.
    (/ (adjusted-ratings header) adjusted-words)))

(defn story-time [header]
  (-
   (/ (System/currentTimeMillis) 1000.)
   (:date_modified
    (first
     (:chapters
      (:story header))))))

(defn likes-per-total-time [header]
  (let [time (story-time header)]
    (/ (adjusted-ratings header) time)))

(defn mature? [header]
  (= "Mature" (:content_rating_text (:story header {}))))

(defn complete? [header]
  (= "Complete" (:status (:story header {}))))

(def only-mature (partial filter mature?))

(def show-mature identity)

(def no-mature (partial filter (comp not mature?)))

(def complete (partial filter complete?))
(def incomplete (partial filter (comp not complete?)))
(def any_status identity)

#+end_src

** Results File Generation

=generate-all-html!= uses each utility function in turn
to sort the list of stories, then produces static HTML files
based on the name of the function. So for =overall-ratings=,
=generate-all-html!= will first sort all the stories by
their total likes, then filter the resulting list by
complete and mature, then take different amounts of results
from the final list and run them through
=pony-headers->html= to generate files named:

#+BEGIN_EXAMPLE
overall-ratings_only-mature_complete_16.html
overall-ratings_only-mature_complete_64.html
overall-ratings_only-mature_complete_512.html
overall-ratings_only-mature_incomplete_16.html
overall-ratings_only-mature_incomplete_64.html
overall-ratings_only-mature_incomplete_512.html
overall-ratings_only-mature_any_status_16.html
overall-ratings_only-mature_any_status_64.html
overall-ratings_only-mature_any_status_512.html
overall-ratings_show-mature_complete_16.html
overall-ratings_show-mature_complete_64.html
overall-ratings_show-mature_complete_512.html
overall-ratings_show-mature_incomplete_16.html
overall-ratings_show-mature_incomplete_64.html
overall-ratings_show-mature_incomplete_512.html
overall-ratings_show-mature_any_status_16.html
overall-ratings_show-mature_any_status_64.html
overall-ratings_show-mature_any_status_512.html
overall-ratings_no-mature_complete_16.html
overall-ratings_no-mature_complete_64.html
overall-ratings_no-mature_complete_512.html
overall-ratings_no-mature_incomplete_16.html
overall-ratings_no-mature_incomplete_64.html
overall-ratings_no-mature_incomplete_512.html
overall-ratings_no-mature_any_status_16.html
overall-ratings_no-mature_any_status_64.html
overall-ratings_no-mature_any_status_512.html
#+END_EXAMPLE

For a total of 27 static files per utility function.

#+name: pony-clj-results
#+begin_src clojure
(in-ns 'org.aurellem.pony)

;; TODO memoize utility and filter fns for greater speed +
;;   elegance

(def ranking-vars [#'likes-per-total-time #'total-comments
                   #'overall-ratings #'ratings-per-word
                   #'ratings-per-view
                   #'ml-rating-function])
(def mature-vars  [#'only-mature #'show-mature #'no-mature])
(def complete-vars [#'complete #'incomplete #'any_status])
(def limits [16 64 512])

(defn generate-all-html!
  ([limit]
     ;; remove old static results files
     (dorun
      (map
       #(.delete %)
       (rest
        (file-seq
         (File. "/home/r/proj/pony-stories/results")))))
     ;; generated last-modified file
     (spit (File. results-root "last-updated.html")
           (date->str (.toDate (Instant.))))
     (let [valid-headers
           (filter
            valid?
            (map pony-header
                 (range (min (pony-header-count) limit))))]
       (dorun 
        (for [ranking-var ranking-vars]
          (let [sorted-headers
                (sort-by
                 (comp - (var-get ranking-var))
                 (map sanitize
                      valid-headers))]
            (dorun
             (for [mature-var  mature-vars
                   complete-var complete-vars]
               (let [final-headers
                     ((var-get complete-var)
                      ((var-get mature-var) sorted-headers))]
                 (dorun
                  (for [limit limits]
                    (let [html
                          (pony-headers->html
                           (take
                            limit
                            final-headers))
                          target
                          (File. results-root
                                 (str 
                                  (clojure.string/join
                                   "_"
                                   [(:name (meta ranking-var))
                                    (:name (meta mature-var))
                                    (:name (meta complete-var))
                                    limit]) ".html"))]
                      (println (.getName target))
                      (spit target html))))))))))))
        ([] (generate-all-html! (pony-header-count))))

(defn redeploy-site []
  (update-header-db!)
  (generate-all-html!))
#+end_src

** Linear Programming

Just for fun, I use the linear programming package [[http://lpsolve.sourceforge.net/5.5/][lpsolve]]
to automatically construct a sixth utility function out of a
linearly weighted combination of the other utility
functions, based on some training data. The clojure
interface to lpsolve which I am using here is described in
[[http://aurellem.org/pokemon-types/html/lpsolve.html][this blog post]].

#+name: pony-clj-lp
#+begin_src clojure
(def story-ratings
  "This is a sequence of [story-id goodness] for selected
  stories."
  [[1888      50] ;; My Little Dashie
   [4656      80] ;; Anthropology'
   [6195      75] ;; It Takes a Village
   [1422      60] ;; Romance Reports
   [755       60] ;; On a Cross And Arrow
   [21583     75] ;; Twilight's List
   [29271     60] ;; Princess Celestia Hates Tea

   [9329      40] ;; Beating the Heat
   [25350     45] ;; Twilight Sparkle Earns the Feature Box
   [18786     65] ;; Field Notes on Alicorn Reproductive Behavior
   [25944     40] ;; Twilight's First Time
   
   [1526      30] ;; Haylo: A New World
   ])

(def pony-dimensions
  [["adjusted ratings" adjusted-ratings    1]
   ["comments"         total-comments      1]
   ["ratings per word"   ratings-per-word  1]
   ["ratings per view"   ratings-per-view  1]
   ])

(defn normalize [vals]
  (let [minimum (apply min vals)
        width (- (apply max vals) (apply min vals))]
    (map (fn [val]
           (float (/ (- val minimum)  width)))
         vals)))

(defn optimize-pony-stories
  ([story-ratings pony-dimensions]
     (let [headers (mapv (comp pony-header first)
                         story-ratings)
           columns
           (vec
            (for [f (map second pony-dimensions)]
              (normalize (map f headers))))
           constraint-matrix
           (vec (for [n (range (count headers))]
                   (mapv #(nth % n) columns)))
           b (mapv second story-ratings)
           c (mapv #(nth % 2) pony-dimensions)]
       
       (clojure.pprint/pprint constraint-matrix)
       (clojure.pprint/pprint b)
       (clojure.pprint/pprint c)

       (lp-solve 
        constraint-matrix b c
        (set-variable-names
         lps (map first pony-dimensions))
         (set-constraints lps LpSolve/LE)
        (.setMaxim lps)
        (results lps))))
  ([] (optimize-pony-stories
       story-ratings pony-dimensions)))

(def ml-rating-function
  (let [linear-coefficients
        (:optimal-values
         (optimize-pony-stories story-ratings
                                pony-dimensions))]
    (comp
     (partial reduce +)
     (apply juxt
            (map (fn [coef f] (comp (partial * coef) f))
                 linear-coefficients
                 (map second pony-dimensions))))))

#+end_src

** Clojure Packages
This is the namespace declaration for all of the clojure
code above.

#+name: pony-clj-header
#+begin_src clojure
(ns org.aurellem.pony
  (:require clojure.data.json)
  (:use hiccup.core)
  (:use clojure.tools.logging)
  (:use pokemon.lpsolve)
  (:import lpsolve.LpSolve)
  (:import java.net.URL java.io.File)
  (:import (org.joda.time Instant Duration)))
#+end_src


* jquery/HTML Frontend

The site is a simple view of static html files that will
have been already generated by the clojure backend.

** javascript defines the actions 
    
The following is some simple javascript that will gather the
four search dimensions from radio buttons, construct the
appropriate static file name, and display the file using
jquery's =load= function whenever the state of any radio
button changes.

#+name: pony-js
#+begin_src javascript
var pony = {};

pony.search_result_target = 
    function (search_alg, mature, complete, num_results) {
	return "results/" + search_alg + 
	    "_" + mature + "_" + complete + 
	    "_" + num_results + ".html";
    };

// get the desired search options from the radio buttons
pony.gather_target = 
    function () {
	
	var search_alg = 
	    $('input[name=search_alg]:checked').val();
	
	var mature = 
	    $('input[name=mature]:checked').val();

	var complete = 
	    $('input[name=complete]:checked').val();
	
	var num_results = 
	    $('input[name=num_results]:checked').val();

	return pony.search_result_target(
	    search_alg, mature, complete, num_results);
    };

pony.last_updated = 
    function () {
	$("#last-updated").load("results/last-updated.html");
    };

pony.update_search_results = 
    function () {
	$("#search-results").load(pony.gather_target());
    };

pony.ajax_init = 
    function () {
	$('input:radio').change(pony.update_search_results);
    };

pony.main_init = 
    function() {
	pony.last_updated();
	pony.update_search_results();
	pony.ajax_init();
    };

$(document).ready(pony.main_init);
#+end_src


** html defines the structure

Now that the interactive portion of the site is defined it
is time to create the overall structure of the website. The
structure of the site is a list of lists --- there are lists
of radio buttons which serve as a controls for the user to
view a list of results. This html defines the radio buttons
which will serve as controls and leaves an empty div for the
list of results, which will be generated by clojure and
loaded by jquery depending on the state of the radio
buttons. The containing divs and ids and classes are so
everything can be styled in css later.


#+name: pony-html
#+begin_src html

<!DOCTYPE html>
<html>

  <head>
    
    <meta charset="utf-8">
    <title>fimfinder.net | fimfiction.net statistics</title>
    <link href="css/pony-stories.css" 
	  rel="stylesheet" type="text/css"/>

    <script type="text/javascript"
	    src="js/jquery-1.7.2.min.js"></script>
    
    <script type="text/javascript"
	    src="js/pony-stories.js"></script> 
  </head>

  <body>
    <div class="header">
      <div class="header-text">
	<h1 class="header">FimFinder</h1>
	Advanced Search for 
	<a class="header" href="http://www.fimfiction.net/">
	  fimfiction.net</a>.
	<br>
	Last Updated : <div id="last-updated"></div>
      </div>
      <a class="header" href="./pony-stories.html">
	<div class="open-source">
	  This Website is Completely Free Software. Click here
	  to see the source.
	</div>
      </a>
    </div>
    
    <div id="search-algorithm-container">
      <ul id="search-algorithm-list">
	


	<li class="search-algorithm">
	  <label for="search2">
	    <div class="search-algorithm" id="search-top-left">
	      <div class="search-desc">
		<input type="radio" id="search2"
		       name="search_alg"
		       value="likes-per-total-time"
		       checked="yes"/>
		Likes/Time
		<p><em>Let the newer stories have a
		    chance...</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/fluttershy.png"/>
	      </div>
	    </div>
	  </label>
	</li>


	<li class="search-algorithm">
	  <label for="search3">
	    <div class="search-algorithm">
	      <div class="search-desc">
		<input type="radio" id="search3"
		       name="search_alg"
		       value="total-comments"/>
		Total Comments
		<p><em>It's a PARTY in the Comments Section!
		</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/pinkie-pie.png"
		     width=40/>
	      </div>
	    </div>
	  </label>
	</li>


	<li class="search-algorithm">
	  <label for="search1">
	    <div class="search-algorithm" id="search-top-right">  
	      <div class="search-desc">
		<input type="radio" id="search1"
		       name="search_alg"
		       value="overall-ratings"/>
		Total Likes
		<p><em>As popular as popular can be!</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/rarity.png"
		     width=40/>
	      </div>

	    </div>
	  </label>
	</li>

	
	<li class="search-algorithm">
	  <label for="search4">
	    <div class="search-algorithm" id="search-bottom-left">
	      <div class="search-desc">
		<input type="radio" id="search4"
		       name="search_alg"
		       value="ratings-per-word"/>
		Likes/Words
		<p><em>Each Word Crafted with Quality.</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/applejack.png"
		     width=40/>
	      </div>
	    </div>
	  </label>
	</li>
	

	<li class="search-algorithm">
	  <label for="search5">
	    <div class="search-algorithm">
	      <div class="search-desc">
		<input type="radio" id="search5"
		       name="search_alg"
		       value="ratings-per-view"/>
		Likes/Views
		<p><em>These Stories are so Awesome!</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/rainbow-dash.png"
		     width=40/>
	      </div>
	    </div>
	  </label>
	</li>


	<li class="search-algorithm">
	  <label for="search6">
	    <div class="search-algorithm" id="search-bottom-right">
	      <div class="search-desc">
		<input type="radio" id="search6" name="search_alg"
		       value="ml-rating-function"/> 
		Combination
		<p><em>Isn't Machine Learning Great?</em></p>
	      </div>		
	      <div class="cutie-mark">
		<img class="cutie-mark"
		     src="../images/twilight-sparkle.png"
		     width="40"/>
	      </div>
	    </div>
	  </label>
	</li>



      </ul>
      
    </div>

    <div id="other-options">

      <div id="mature-options">
	<!-- Mature: -->
	<ul>
	  <li>
	    <div class="input-choice">
	      <label for="mature_1">
		<input type="radio" name="mature" 
		       checked="yes"
		       id="mature_1" value="no-mature"/>
		Non-Mature
	      </label>
	    </div>
	  </li>
	  <li>
	    <div class="input-choice">
	      <label for="mature_3">
		<input type="radio" name="mature" 
		       id="mature_3" value="only-mature"/>
		Mature
	      </label>
	    </div>
	  </li>
	  

	  <li>
	    <div class="input-choice">
	      <label for="mature_2">
		<input type="radio" name="mature" 
		       id="mature_2" value="show-mature"/>
		Both
	      </label>
	    </div>
	  </li>
	  
	</ul>
      </div>

      <div id="complete-options">
	<!-- Results: -->
	<ul>
	  <li>
	    <div class="input-choice">
	      <label for="complete_1">
		<input type="radio" name="complete" 
		       id="complete_1" value="complete"/>
		Complete
	      </label>
	    </div>
	  </li>
	  <li>
	    <div class="input-choice">
	      <label for="complete_2">
		<input type="radio" name="complete" 
		       id="complete_2" value="incomplete"/>
		Not Complete
	      </label>
	    </div>
	  </li>
	  <li>
	    <div class="input-choice">
	      <label for="complete_3">
		<input type="radio" name="complete" 
		       checked="yes"
		       id="complete_3" value="any_status"/>
		Both
	      </label>
	    </div>
	  </li>
	</ul>
      </div>
      <div id="num_results_options">
	<!-- Results: -->
	<ul>
	  <li>
	    <div class="input-choice">
	      <label for="num_results_1">
		<input type="radio" name="num_results" 
		       checked="yes"
		       id="num_results_1" value="16"/>
		16
	      </label>
	    </div>
	  </li>
	  <li>
	    <div class="input-choice">
	      <label for="num_results_2">
		<input type="radio" name="num_results" 
		       id="num_results_2" value="64"/>
		64
	      </label>
	    </div>
	  </li>
	  <li>
	    <div class="input-choice">
	      <label for="num_results_3">
		<input type="radio" name="num_results" 
		       id="num_results_3" value="512"/>
		512
	      </label>
	    </div>
	  </li>
	</ul>
      </div>
    </div>
    
    <div id="search-results">
      <!-- This section will be replaced by 
	   actual results from the clojure backend. -->
      no results so far!
    </div>

  </body>
</html>
#+end_src


** css defines the style

I decided to go with a theme with bright colors that mimic
the colors of the six main ponies, with a special emphasis
on Twilight Sparkle since the site serves as a sort of card
catalog for the library of stories on
fimfiction.net. Thus, the title bar is purple and white
while the secondary control elements are the colors of the
other ponies. For the search results I was trying to go for
a library-card catalog feel, thus the yellow color and
loose organization of story statistics.x Of note here is the
tabs which appear at the left of each search result --- they
change color depending on whether that particular story has
been visited, helping the user to find stories they have not
read before.

#+name:pony-css
#+begin_src java

/* Header */
body{
    margin:0em;
    padding:0em;
    font: "Georgia Serif", Times, Palatino;
}

div{
    padding:0px;
    margin:0px;
    overflow:hidden;
}


h1.header{
    margin:0em;
}

a.header{
    color:white;
}

div.header-text{
    padding-left:4.375em;
    padding-bottom:1.5625em;
    padding-top:1.5625em;
    font-size:1.25em;    
    font-weight:bold;
    float:left;
}

div.header{
    padding:0em;
    margin:0em;
    margin-bottom:3.125em;
    background-color:#ae48c8;
    color:white;
    border-bottom-style:solid;
    border-bottom-width: 1.875em;
    border-bottom-color:#f584ff;
}

div.open-source{
    float:right;
    width:10em;
    margin:0em;
    padding:1.5625em;
    margin-right:1em;
}

div#last-updated{
    display:inline;
}

/* Search Controls */

hr.search{
    background-color:white;
    padding:0em;
    margin:0em;
    height:0.125em;
    color:white;
    border:0em;
}


ul{
    list-style-type:none;
    padding:0em;
    margin: 0em;
}


ul.search-algorithm-list{
}

li.search-algorithm{
    float:left;
    display:inline;
}

div#search-algorithm-container{
    /*border-radius: 1em;
    -moz-border-radius: 1em;*/
    background-color:white;
    /*#665577;*/
    color:white;
    width:67em;
    margin-left:auto;
    margin-right:auto;
}

div.search-algorithm{
    border-style: none;
    background-color:#665577;
    overflow:hidden;
    width:20em;
    height:6em;
    padding:1em;
    border:0.0625em solid white;
}

div.search-desc{
    float:left;
    width:15em;
}

div.cutie-mark{
    float:left;
    width:2.5em;
    margin:1em;
}

img.cutie-mark{
    display:inline;
    width:2.5em;
    height:3.6em;
}

div.search-algorithm:hover{
    background-color:#9977aa;
}

div#search-top-left{
    border-top-left-radius:1em;
    -moz-border-radius-topleft:1em;
}

div#search-top-right{
    border-top-right-radius:1em;	
    -moz-border-radius-topright:1em;
}

div#search-bottom-left{
    border-bottom-left-radius:1em;
    -moz-border-radius-bottomleft:1em;

}

div#search-bottom-right{
    border-bottom-right-radius:1em;
    -moz-border-radius-bottomright:1em;
}

div#other-options{
    width:31.875em;

    margin-left:auto;
    margin-right:auto;
}

div#num_results_options{
    background-color:#a5eeff;
    -moz-border-radius: 1em;
    border-radius: 1em;
    border-style: solid;
    border-width: 0.0625em;
    padding: 0.625em;
    width:6.25em;
    float:left;
    margin-top:0.625em;
    margin-right:0.3125em;
}

div#mature-options{
    background-color:#a5eeff;
    -moz-border-radius: 1em;
    border-radius: 1em;
    border-style: solid;
    border-width: 0.0625em;
    padding: 0.625em;
    width:10em;
    float:left;
    margin-top:0.625em;
    margin-right:0.3125em;
}

div#complete-options{
    background-color:#a5eeff;
    -moz-border-radius: 1em;
    border-radius: 1em;
    border-style: solid;
    border-width: 0.0625em;
    padding: 0.625em;
    width:10em;
    float:left;
    margin-top:0.625em;
    margin-right:0.3125em;
}


/* Results */

/* Circled Number on at the Right of each Story Result */
div.pony-index{
    float:right;
    -moz-border-radius: 3.75em;
    border-radius: 3.75em;
    border-style: solid;
    border-width: 0.0625em;
    width:3.75em;
    height:3.75em;
    margin-left:1.875em;
    text-align:center;
    margin-top:1.875em;
}

div.inner-index{
    margin-top:1.25em;
    margin-bottom:auto;
}

ol.pony-list{
    list-style-type:none;
    margin:0em;
    padding:0em;
}

li.pony-list-item{
    width:40.625em;
}

div.pony-description{
    width:40em;
    margin-top:0.4375em;
}

div.pony-info{
    margin-top:1em;
    float:left;
}

div.pony-summary{
    float:left;
    width:14.375em;
}

div.pony-stats{
    float:left;
    width:12.1875em;
}

div.pony-stats-desc {
    width:5.9375em;
    float:left;
}

div.pony-stats-value {
    display:inline;
}

div.pony-categories{
    float:left;
    width:7.5em;
}

em.pony-title {
    font-size:1.875em;
    font-style:normal;
    font-variant:small-caps;
}

em.pony-author {
    font-size:1.125em;
}

em.pony-by{
    font-style:normal;
}

ul.pony-summary{
    list-style-type:none;
}

ul.pony-categories{
    list-style-type:none;
}

ul.pony-stats{
    list-style-type:none;
}

a.pony-link{
    text-decoration:none;
    display:block;
    /*background-color:#000000;*/
    -moz-border-radius: 1.875em;
    border-radius: 1.875em;
    color:black;
}

a.pony-link:visited{
    background-color:#BBB;
    color:#333;
}

div#search-results{
    width:44.6875em;
    margin-left:auto;
    margin-right:auto;
}

div.pony-list-item{
    width:40.625em;
    /*http://www.quirksmode.org/css/clearing.html*/
    /*background-color:#d4ffa5;*/
    background-color:#fdffce;
    -moz-border-radius: 1em;
    border-radius: 1em;
    border-style: solid;
    border-width: 0.0625em;
    margin: 1.25em;
    padding: 1.25em;
}

div.pony-list-item:hover{
    background-color:#fcffaf;
    /*background-color:#91ff30;*/
}
#+end_src


* Upload Script
Finally, a little upload script to put the site on my
server.

#+name: pony-upload
#+begin_src sh
#!/bin/sh

rsync -avz --delete /home/r/proj/pony-stories/html \
                    r@fimfinder.net://home/r/proj/pony-stories

rsync -avz --delete /home/r/proj/pony-stories/images \
                    r@fimfinder.net://home/r/proj/pony-stories

rsync -avz --delete /home/r/proj/pony-stories/org \
                    r@fimfinder.net://home/r/proj/pony-stories

#+end_src


* Links

Overall this was a fun exercise in making a cute little
website involving only static content that looks like it's
dynamic. Most of my time actually ended up in the css since
that's the area where I have the least skill.

Here's some more links you may find interesting:

- [[http://fimfinder.net][the actual website]]
#+html: <ul> <li> <a href="./pony-stories.org">this org file</a> </li> </ul>
- [[https://github.com/weavejester/hiccup/][hiccup]] -- a library for generating html from clojure
- [[http://lpsolve.sourceforge.net/5.5/][lpsolve]] -- open source linear programming library
- [[http://aurellem.org/pokemon-types/html/lpsolve.html][lpsolve for clojure]] -- library for interfacing lpsolve to clojure
- [[http://www.fimfiction.net/][fimfiction.net]] 

  
* COMMENT generate source

#+begin_src css :tangle ../html/css/pony-stories.css
<<pony-css>>
#+end_src

#+begin_src clojure :tangle ../backend/src/org/aurellem/pony.clj
<<pony-clj-header>>
<<pony-clj-db>>
<<pony-clj-html>>
<<pony-clj-utility>>
<<pony-clj-lp>>
<<pony-clj-results>>
#+end_src

#+begin_src html :tangle ../html/search.html
<<pony-html>>
#+end_src

#+begin_src javascript :tangle ../html/js/pony-stories.js
<<pony-js>>
#+end_src

#+begin_src sh :tangle ../upload-ponies.sh
<<pony-upload>>
#+end_src


* COMMENT TODO now [5/5]
- [X] style headings and other interface elements
- [X] complete clojure function to update db
- [X] add "show complete" settings
- [X] add main description
- [X] add title bar with access to org source and search page

* COMMENT TODO later [4/8]
- [X] add numbers to each list entry
- [ ] change 10, 50, 200, to 16, 64, 512
- [X] add some more search utility functions
- [X] orgify everything
- [X] complete multidimensional search
- [ ] decide on name; buy domain name
  (fimfinder,fimfiction-stats, elements of fanfiction)
- [ ] leinengenify everything
- [ ] clean up CSS !!!