FAQ
This is more of a what do I need to do question - but very Go-specific.

Assume a html-template which contains this function:

img src="{{statichash file.png}}"

statichash will return for the parameter file.png --> file.HASH.png, so the
(Go) server will eventually hit a request for file.HASH.png. At the server
side, for performance reasons, the HASH will either be precomputed and fed
to the cache or calculated upon first request (by an external "file
watcher" receiving ioctls) and then added to the cache. A very long
expiration date will be added for browser caching.

Now I wonder what to do when a request like file.HASH.png this hits the
static handler.

If dived into the source of net/http/fs.go. Am I right that in order to
implement this I have to mimic servefile? Like

http.Handle("/static/", hashStaticHandler(statictimeout,
http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

hashStaticHandler will perform:

* parse the request of file.HASH.png and extract the hash-part (if any)
- validate the hash against the cached value of the hash of file.png (this
is an implementation detail, I could calculate the hash upon every request
but that gets computationally heavy)
- IFequal: rewrite the request to file.png and continue serving with
w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
must-revalidate, proxy-revalidate", seconds)) // Update the time-out
h.ServeHTTP(w, r)
- else response with moved permanently and return file.NEWHASH.png

Is this feasible this way?

--

Search Discussions

  • Patrick Mylund Nielsen at Dec 26, 2012 at 12:04 am
    I would use a version number instead -- no real reason to have a complete
    digest for every file. When the file changes, increment foo.n.png, and
    rewrite if n is less than the current. Or use seconds since epoch.

    I usually have a short max-age on the page linking to whatever images, then
    change the hrefs on the page if the file changes, so my static server
    doesn't need to do much/any rewriting.

    On Wed, Dec 26, 2012 at 12:56 AM, Johann Höchtl wrote:

    This is more of a what do I need to do question - but very Go-specific.

    Assume a html-template which contains this function:

    img src="{{statichash file.png}}"

    statichash will return for the parameter file.png --> file.HASH.png, so
    the (Go) server will eventually hit a request for file.HASH.png. At the
    server side, for performance reasons, the HASH will either be precomputed
    and fed to the cache or calculated upon first request (by an external "file
    watcher" receiving ioctls) and then added to the cache. A very long
    expiration date will be added for browser caching.

    Now I wonder what to do when a request like file.HASH.png this hits the
    static handler.

    If dived into the source of net/http/fs.go. Am I right that in order to
    implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout,
    http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

    hashStaticHandler will perform:

    * parse the request of file.HASH.png and extract the hash-part (if any)
    - validate the hash against the cached value of the hash of file.png
    (this is an implementation detail, I could calculate the hash upon every
    request but that gets computationally heavy)
    - IFequal: rewrite the request to file.png and continue serving with
    w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
    must-revalidate, proxy-revalidate", seconds)) // Update the time-out
    h.ServeHTTP(w, r)
    - else response with moved permanently and return file.NEWHASH.png

    Is this feasible this way?

    --

    --
  • Johann Höchtl at Dec 26, 2012 at 7:21 am
    Now that was to fast for me. I understand the concept behind version numbers (and can't remember their disadvantages right now) but what would you keep for what reason on every page? Pls elaborate at bit.

    (Sent from smart phone, can't properly cite)

    --
  • Rodrigo Kochenburger at Dec 26, 2012 at 2:35 am
    Why not pass the the last modification timestamp as hash and as a parameter? I.e. file.png?2012215183500

    It will work out of the box ;)

    --
  • Johann Höchtl at Dec 26, 2012 at 7:11 am
    Parameters are not proxy-server nor CDN-cache friendly. Will work for the browser cache, but thats just one "cache-point"

    --
  • Rodrigo Kochenburger at Dec 26, 2012 at 10:57 am
    You're right. It does not work with all caches.

    On your proposal, if you precompute the file and actually store it on the
    filesystem, you can just server the file with a straight FileServer.
    On Tuesday, December 25, 2012 11:11:12 PM UTC-8, Johann Höchtl wrote:

    Parameters are not proxy-server nor CDN-cache friendly. Will work for the
    browser cache, but thats just one "cache-point"
    --
  • Hraban Luyat at Dec 26, 2012 at 7:06 pm
    Hi,

    It might help to have a close look at your requirements and see if you can
    cut corners, there. Two easier solutions that are not as functionally
    complete but that will do the job:

    - Pre-compute the hashes, create symlinks with the new names (e.g. make
    fingerprints)
    - Put your app behind a webserver and create a URL matching rule for hashes
    that strips the hash and serves the bare file

    If you wrap all references to static files in a function (as you do now) it
    will make it easier to also change the host they are served from: {{cdn
    "myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for
    example. Even if you do not have a CDN, this will make it easy to use a
    webserver only for your static files, which, in turn, makes dealing with
    this specific case much easier.

    The nice thing about using hashes is that it relies solely on the contents
    of the file. Timestamps or versioning requires extra bookkeeping and can
    get confusing if you have multiple locations you serve your static content
    from.

    If possible, try to add an extra hint in the filename, not just the hash.
    E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This
    helps identifying the hash when you have just the requested resource.
    Useful when using regular expressions for URL rewrite rules, or find rules.

    Greetings,

    Hraban

    Τη Τετάρτη, 26 Δεκεμβρίου 2012 12:56:03 π.μ. UTC+1, ο χρήστης Johann Höchtl
    έγραψε:
    This is more of a what do I need to do question - but very Go-specific.

    Assume a html-template which contains this function:

    img src="{{statichash file.png}}"

    statichash will return for the parameter file.png --> file.HASH.png, so
    the (Go) server will eventually hit a request for file.HASH.png. At the
    server side, for performance reasons, the HASH will either be precomputed
    and fed to the cache or calculated upon first request (by an external "file
    watcher" receiving ioctls) and then added to the cache. A very long
    expiration date will be added for browser caching.

    Now I wonder what to do when a request like file.HASH.png this hits the
    static handler.

    If dived into the source of net/http/fs.go. Am I right that in order to
    implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout,
    http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

    hashStaticHandler will perform:

    * parse the request of file.HASH.png and extract the hash-part (if any)
    - validate the hash against the cached value of the hash of file.png
    (this is an implementation detail, I could calculate the hash upon every
    request but that gets computationally heavy)
    - IFequal: rewrite the request to file.png and continue serving with
    w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
    must-revalidate, proxy-revalidate", seconds)) // Update the time-out
    h.ServeHTTP(w, r)
    - else response with moved permanently and return file.NEWHASH.png

    Is this feasible this way?
    --
  • Johann Höchtl at Dec 27, 2012 at 10:03 am

    Am Mittwoch, 26. Dezember 2012 16:12:56 UTC+1 schrieb Hraban Luyat:
    Hi,

    It might help to have a close look at your requirements and see if you can
    cut corners, there. Two easier solutions that are not as functionally
    complete but that will do the job:

    - Pre-compute the hashes, create symlinks with the new names (e.g. make
    fingerprints)
    That sounds like a very good idea, I will investigate in that direction.

    - Put your app behind a webserver and create a URL matching rule for hashes
    that strips the hash and serves the bare file

    If you wrap all references to static files in a function (as you do now)
    it will make it easier to also change the host they are served from: {{cdn
    "myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for
    example. Even if you do not have a CDN, this will make it easy to use a
    webserver only for your static files, which, in turn, makes dealing with
    this specific case much easier.

    The nice thing about using hashes is that it relies solely on the contents
    of the file. Timestamps or versioning requires extra bookkeeping and can
    get confusing if you have multiple locations you serve your static content
    from.

    If possible, try to add an extra hint in the filename, not just the hash.
    E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This
    helps identifying the hash when you have just the requested resource.
    Useful when using regular expressions for URL rewrite rules, or find rules.
    That are all very good advices, thank you!

    With the symlink solution I can leave the StaticFileServer as is and wrap
    the logic into the template static handler which will do a filename -->
    hasedfilename handle.

    Greetings,

    Hraban

    Τη Τετάρτη, 26 Δεκεμβρίου 2012 12:56:03 π.μ. UTC+1, ο χρήστης Johann
    Höchtl έγραψε:
    This is more of a what do I need to do question - but very Go-specific.

    Assume a html-template which contains this function:

    img src="{{statichash file.png}}"

    statichash will return for the parameter file.png --> file.HASH.png, so
    the (Go) server will eventually hit a request for file.HASH.png. At the
    server side, for performance reasons, the HASH will either be precomputed
    and fed to the cache or calculated upon first request (by an external "file
    watcher" receiving ioctls) and then added to the cache. A very long
    expiration date will be added for browser caching.

    Now I wonder what to do when a request like file.HASH.png this hits the
    static handler.

    If dived into the source of net/http/fs.go. Am I right that in order to
    implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout,
    http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

    hashStaticHandler will perform:

    * parse the request of file.HASH.png and extract the hash-part (if any)
    - validate the hash against the cached value of the hash of file.png
    (this is an implementation detail, I could calculate the hash upon every
    request but that gets computationally heavy)
    - IFequal: rewrite the request to file.png and continue serving with
    w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
    must-revalidate, proxy-revalidate", seconds)) // Update the time-out
    h.ServeHTTP(w, r)
    - else response with moved permanently and return file.NEWHASH.png

    Is this feasible this way?
    --
  • Hraban Luyat at Dec 30, 2012 at 10:16 am
    Hi,

    Here's a script I use to generate md5 fingerprinted names, md5name.sh:

    #!/bin/bash

    if [[ ! -f "$1" ]]
    then
    echo "Usage: $0 <filename>" >&2
    echo >&2
    echo "File must exist and be a standard file" >&2
    exit 1
    fi

    h=`md5sum "$1" | awk '{ print $1 }'`
    d=`dirname "$1"`
    b=`basename "$1"`
    e=""
    if [[ "$b" == *.* ]]
    then
    e=".${b##*.}"
    b="${b%.*}"
    fi
    echo "$d/$b-md5-$h$e"

    I am not sure what would happen with filenames with spaces.

    Here are the relevant parts of a Makefile (make fingerprints):

    # Watch out for filenames with spaces
    STATICS := $(shell find static/ -type f -and \! \( -name '.*' -o -name '*~' \))
    # Use GNU Make friendly files to annotate when the latest fingerprint was made
    FINGERHINTS := $(patsubst %,makegarbage/fingerprints/%,$(STATICS))

    $(FINGERHINTS): makegarbage/fingerprints/%: %
    mkdir -p $(shell dirname $@)
    ln -s -f $(shell basename $<) `./md5name.sh $<`
    touch $@

    fingerprints: $(FINGERHINTS)

    clean-fingerprints:
    rm -rf makegarbage/fingerprints
    find static -name '*-md5-*' -delete
    CLEANERS += clean-fingerprints

    clean: $(CLEANERS)
    rm -rf makegarbage

    I succombed to using a bookkeeping directory, "makegarbage", where I
    keep track of when a fingerprint was created for each file. It is easy
    to use, easy to clean (rm -rf makegarbage) and it leaves no noise in
    the directory of the static files themselves.

    To clear all old fingerprints, make clean does the job (or make
    clean-fingerprints). To update the newest fingerprints, make
    fingerprints (or add it as a dependency for the rest of the program).

    The caveat here is that old fingerprints will point to new files,
    until purged. There is a benefit to this: you can test changes by
    refreshing the page without having to regenerate the fingerprints and
    restart the server every time.

    If somebody knows of a nice way to improve this workflow I would be
    happy to hear about it.

    Greetings,

    Hraban

    On Thu, Dec 27, 2012 at 11:03 AM, Johann Höchtl
    wrote:

    Am Mittwoch, 26. Dezember 2012 16:12:56 UTC+1 schrieb Hraban Luyat:
    Hi,

    It might help to have a close look at your requirements and see if you can
    cut corners, there. Two easier solutions that are not as functionally
    complete but that will do the job:

    - Pre-compute the hashes, create symlinks with the new names (e.g. make
    fingerprints)

    That sounds like a very good idea, I will investigate in that direction.
    - Put your app behind a webserver and create a URL matching rule for
    hashes that strips the hash and serves the bare file

    If you wrap all references to static files in a function (as you do now)
    it will make it easier to also change the host they are served from: {{cdn
    "myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for example.
    Even if you do not have a CDN, this will make it easy to use a webserver
    only for your static files, which, in turn, makes dealing with this specific
    case much easier.

    The nice thing about using hashes is that it relies solely on the contents
    of the file. Timestamps or versioning requires extra bookkeeping and can get
    confusing if you have multiple locations you serve your static content from.

    If possible, try to add an extra hint in the filename, not just the hash.
    E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This
    helps identifying the hash when you have just the requested resource. Useful
    when using regular expressions for URL rewrite rules, or find rules.

    That are all very good advices, thank you!

    With the symlink solution I can leave the StaticFileServer as is and wrap
    the logic into the template static handler which will do a filename -->
    hasedfilename handle.

    Greetings,

    Hraban

    Τη Τετάρτη, 26 Δεκεμβρίου 2012 12:56:03 π.μ. UTC+1, ο χρήστης Johann
    Höchtl έγραψε:
    This is more of a what do I need to do question - but very Go-specific.

    Assume a html-template which contains this function:

    img src="{{statichash file.png}}"

    statichash will return for the parameter file.png --> file.HASH.png, so
    the (Go) server will eventually hit a request for file.HASH.png. At the
    server side, for performance reasons, the HASH will either be precomputed
    and fed to the cache or calculated upon first request (by an external "file
    watcher" receiving ioctls) and then added to the cache. A very long
    expiration date will be added for browser caching.

    Now I wonder what to do when a request like file.HASH.png this hits the
    static handler.

    If dived into the source of net/http/fs.go. Am I right that in order to
    implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout,
    http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

    hashStaticHandler will perform:

    * parse the request of file.HASH.png and extract the hash-part (if any)
    - validate the hash against the cached value of the hash of file.png
    (this is an implementation detail, I could calculate the hash upon every
    request but that gets computationally heavy)
    - IFequal: rewrite the request to file.png and continue serving with
    w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
    must-revalidate, proxy-revalidate", seconds)) // Update the time-out
    h.ServeHTTP(w, r)
    - else response with moved permanently and return file.NEWHASH.png

    Is this feasible this way?
    --
    --
  • Johann Höchtl at Jan 4, 2013 at 8:55 pm

    On 12/29/2012 04:12 PM, Hraban Luyat wrote:
    Hi,

    Here's a script I use to generate md5 fingerprinted names, md5name.sh:

    #!/bin/bash

    if [[ ! -f "$1" ]]
    then
    echo "Usage: $0 <filename>" >&2
    echo >&2
    echo "File must exist and be a standard file" >&2
    exit 1
    fi

    h=`md5sum "$1" | awk '{ print $1 }'`
    d=`dirname "$1"`
    b=`basename "$1"`
    e=""
    if [[ "$b" == *.* ]]
    then
    e=".${b##*.}"
    b="${b%.*}"
    fi
    echo "$d/$b-md5-$h$e"

    I am not sure what would happen with filenames with spaces.

    Here are the relevant parts of a Makefile (make fingerprints):

    # Watch out for filenames with spaces
    STATICS := $(shell find static/ -type f -and \! \( -name '.*' -o -name '*~' \))
    # Use GNU Make friendly files to annotate when the latest fingerprint was made
    FINGERHINTS := $(patsubst %,makegarbage/fingerprints/%,$(STATICS))

    $(FINGERHINTS): makegarbage/fingerprints/%: %
    mkdir -p $(shell dirname $@)
    ln -s -f $(shell basename $<) `./md5name.sh $<`
    touch $@

    fingerprints: $(FINGERHINTS)

    clean-fingerprints:
    rm -rf makegarbage/fingerprints
    find static -name '*-md5-*' -delete
    CLEANERS += clean-fingerprints

    clean: $(CLEANERS)
    rm -rf makegarbage

    I succombed to using a bookkeeping directory, "makegarbage", where I
    keep track of when a fingerprint was created for each file. It is easy
    to use, easy to clean (rm -rf makegarbage) and it leaves no noise in
    the directory of the static files themselves.

    To clear all old fingerprints, make clean does the job (or make
    clean-fingerprints). To update the newest fingerprints, make
    fingerprints (or add it as a dependency for the rest of the program).
    Thank you for this tips ... I always enjoy seeing some shell scripting
    which is (almost) all wizardry to me.

    Johann
    The caveat here is that old fingerprints will point to new files,
    until purged. There is a benefit to this: you can test changes by
    refreshing the page without having to regenerate the fingerprints and
    restart the server every time.

    If somebody knows of a nice way to improve this workflow I would be
    happy to hear about it.

    Greetings,

    Hraban

    On Thu, Dec 27, 2012 at 11:03 AM, Johann Höchtl
    wrote:

    Am Mittwoch, 26. Dezember 2012 16:12:56 UTC+1 schrieb Hraban Luyat:
    Hi,

    It might help to have a close look at your requirements and see if you can
    cut corners, there. Two easier solutions that are not as functionally
    complete but that will do the job:

    - Pre-compute the hashes, create symlinks with the new names (e.g. make
    fingerprints)

    That sounds like a very good idea, I will investigate in that direction.
    - Put your app behind a webserver and create a URL matching rule for
    hashes that strips the hash and serves the bare file

    If you wrap all references to static files in a function (as you do now)
    it will make it easier to also change the host they are served from: {{cdn
    "myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for example.
    Even if you do not have a CDN, this will make it easy to use a webserver
    only for your static files, which, in turn, makes dealing with this specific
    case much easier.

    The nice thing about using hashes is that it relies solely on the contents
    of the file. Timestamps or versioning requires extra bookkeeping and can get
    confusing if you have multiple locations you serve your static content from.

    If possible, try to add an extra hint in the filename, not just the hash.
    E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This
    helps identifying the hash when you have just the requested resource. Useful
    when using regular expressions for URL rewrite rules, or find rules.

    That are all very good advices, thank you!

    With the symlink solution I can leave the StaticFileServer as is and wrap
    the logic into the template static handler which will do a filename -->
    hasedfilename handle.

    Greetings,

    Hraban

    Τη Τετάρτη, 26 Δεκεμβρίου 2012 12:56:03 π.μ. UTC+1, ο χρήστης Johann
    Höchtl έγραψε:
    This is more of a what do I need to do question - but very Go-specific.

    Assume a html-template which contains this function:

    img src="{{statichash file.png}}"

    statichash will return for the parameter file.png --> file.HASH.png, so
    the (Go) server will eventually hit a request for file.HASH.png. At the
    server side, for performance reasons, the HASH will either be precomputed
    and fed to the cache or calculated upon first request (by an external "file
    watcher" receiving ioctls) and then added to the cache. A very long
    expiration date will be added for browser caching.

    Now I wonder what to do when a request like file.HASH.png this hits the
    static handler.

    If dived into the source of net/http/fs.go. Am I right that in order to
    implement this I have to mimic servefile? Like

    http.Handle("/static/", hashStaticHandler(statictimeout,
    http.StripPrefix("/static/", http.FileServer(http.Dir("static")))))

    hashStaticHandler will perform:

    * parse the request of file.HASH.png and extract the hash-part (if any)
    - validate the hash against the cached value of the hash of file.png
    (this is an implementation detail, I could calculate the hash upon every
    request but that gets computationally heavy)
    - IFequal: rewrite the request to file.png and continue serving with
    w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public,
    must-revalidate, proxy-revalidate", seconds)) // Update the time-out
    h.ServeHTTP(w, r)
    - else response with moved permanently and return file.NEWHASH.png

    Is this feasible this way?
    --
    --
  • Martin Angers at Dec 29, 2012 at 4:28 pm
    Unless the goal is to experiment with go to build this type of mechanism, I would recommend using Varnish in front of your Go web server to handle the caching. That's what it's for, and although I'm no Varnsih expert, I believe it handles your use-case quite easily.

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 25, '12 at 11:56p
activeJan 4, '13 at 8:55p
posts11
users5
websitegolang.org

People

Translate

site design / logo © 2021 Grokbase