[go-nuts] net/http: Serving static files with hash part (cache and proxy-friendly)

I would use a version number instead -- no real reason to have a complete
digest for every file. When the file changes, increment foo.n.png, and
rewrite if n is less than the current. Or use seconds since epoch.

I usually have a short max-age on the page linking to whatever images, then
change the hrefs on the page if the file changes, so my static server
doesn't need to do much/any rewriting.

Now that was to fast for me. I understand the concept behind version numbers (and can't remember their disadvantages right now) but what would you keep for what reason on every page? Pls elaborate at bit.

(Sent from smart phone, can't properly cite)

--

Why not pass the the last modification timestamp as hash and as a parameter? I.e. file.png?2012215183500

It will work out of the box ;)

--

Parameters are not proxy-server nor CDN-cache friendly. Will work for the browser cache, but thats just one "cache-point"

--

You're right. It does not work with all caches.

On your proposal, if you precompute the file and actually store it on the
filesystem, you can just server the file with a straight FileServer.

Hi,

It might help to have a close look at your requirements and see if you can
cut corners, there. Two easier solutions that are not as functionally
complete but that will do the job:

- Pre-compute the hashes, create symlinks with the new names (e.g. make
fingerprints)
- Put your app behind a webserver and create a URL matching rule for hashes
that strips the hash and serves the bare file

If you wrap all references to static files in a function (as you do now) it
will make it easier to also change the host they are served from: {{cdn
"myfile.png"}} -> //blabla.cloudfront.net/myfile-somehash.png", for
example. Even if you do not have a CDN, this will make it easy to use a
webserver only for your static files, which, in turn, makes dealing with
this specific case much easier.

The nice thing about using hashes is that it relies solely on the contents
of the file. Timestamps or versioning requires extra bookkeeping and can
get confusing if you have multiple locations you serve your static content
from.

If possible, try to add an extra hint in the filename, not just the hash.
E.g.: myfile.png -> myfile-md5-d41d8cd98f00b204e9800998ecf8427e.png. This
helps identifying the hash when you have just the requested resource.
Useful when using regular expressions for URL rewrite rules, or find rules.

Greetings,

Hraban

Τη Τετάρτη, 26 Δεκεμβρίου 2012 12:56:03 π.μ. UTC+1, ο χρήστης Johann Höchtl
έγραψε:

That sounds like a very good idea, I will investigate in that direction.

- Put your app behind a webserver and create a URL matching rule for hashes
That are all very good advices, thank you!

With the symlink solution I can leave the StaticFileServer as is and wrap
the logic into the template static handler which will do a filename -->
hasedfilename handle.

Hi,

Here's a script I use to generate md5 fingerprinted names, md5name.sh:

#!/bin/bash

if [[ ! -f "$1" ]]
then
echo "Usage: $0 <filename>" >&2
echo >&2
echo "File must exist and be a standard file" >&2
exit 1
fi

h=`md5sum "$1" | awk '{ print $1 }'`
d=`dirname "$1"`
b=`basename "$1"`
e=""
if [[ "$b" == *.* ]]
then
e=".${b##*.}"
b="${b%.*}"
fi
echo "$d/$b-md5-$h$e"

I am not sure what would happen with filenames with spaces.

Here are the relevant parts of a Makefile (make fingerprints):

# Watch out for filenames with spaces
STATICS := $(shell find static/ -type f -and \! \( -name '.*' -o -name '*~' \))
# Use GNU Make friendly files to annotate when the latest fingerprint was made
FINGERHINTS := $(patsubst %,makegarbage/fingerprints/%,$(STATICS))

$(FINGERHINTS): makegarbage/fingerprints/%: %
mkdir -p $(shell dirname $@)
ln -s -f $(shell basename $<) `./md5name.sh $<`
touch $@

fingerprints: $(FINGERHINTS)

clean-fingerprints:
rm -rf makegarbage/fingerprints
find static -name '*-md5-*' -delete
CLEANERS += clean-fingerprints

clean: $(CLEANERS)
rm -rf makegarbage

I succombed to using a bookkeeping directory, "makegarbage", where I
keep track of when a fingerprint was created for each file. It is easy
to use, easy to clean (rm -rf makegarbage) and it leaves no noise in
the directory of the static files themselves.

To clear all old fingerprints, make clean does the job (or make
clean-fingerprints). To update the newest fingerprints, make
fingerprints (or add it as a dependency for the rest of the program).

The caveat here is that old fingerprints will point to new files,
until purged. There is a benefit to this: you can test changes by
refreshing the page without having to regenerate the fingerprints and
restart the server every time.

If somebody knows of a nice way to improve this workflow I would be
happy to hear about it.

Greetings,

Hraban

On Thu, Dec 27, 2012 at 11:03 AM, Johann Höchtl
wrote:

Thank you for this tips ... I always enjoy seeing some shell scripting
which is (almost) all wizardry to me.

Johann

Unless the goal is to experiment with go to build this type of mechanism, I would recommend using Varnish in front of your Go web server to handle the caching. That's what it's for, and although I'm no Varnsih expert, I believe it handles your use-case quite easily.

--