[Puppet Users] Advice on Puppet update to 4

Greetings,

First, a short intro. :-)

Last year I invested quite a bit of time building out a puppet
infrastructure. I have +150 Scientific Linux 6 and RHEL 6 servers running
puppet 3.7.1. It has been working really well concidering that I basically
haven't touched it since last October....

The issues I see are things like:
* I never got around to moving off of WEBrick for the webserver. So the box
takes a beating (it is always under a lot of load with that many clients).
* I haven't gotten around to updating past 3.7.1....that is kinda a big
one...
* A lot of the poorly written modules _I_ wrote a year ago can now be
easily replaced with much better versions from the forge.

Basically, things are pretty much my fault for not properly caring for it.
:-D

But! It is working /really/ well. I don't have many changes except the
addition/decomission of servers. I keep meaning to spend time, but it
hasn't happened....until last weekend when the puppetmaster server went off
line...I have the server functioning again, but the motherboard is *not*
long for this world.

Since I am rebuilding anyway and I _know_ that our infrastructure is about
to grow another 100 servers in the next few months, I figured I should look
into upgrading to puppet 4.1. I have read through the changelog (didn't
understand a lot, but I recognize there are a lot of changes I will have to
make). If I am willing to take the time to migrate to puppet-forge modules
anyway (plus hopefully fix any bad code I can't replace), wouldn't this be
a good time to switch puppet versions too?

Questions!

1a)
I am debating as to whether I need to build multiple puppet masters or not.
Is the following documentation still valid for the 4.x series of puppet?
https://docs.puppetlabs.com/guides/scaling_multiple_masters.html

I have a lot of load, a lot of puppet clients, and I wouldn't mind the
reliability of multiple systems. But I am not sure if it is worth the
effort or not. Are there good metrics for when to load balance or not?

I never got around to setting up puppet dashboard, but I would like to. Any
load-balanced solution I would prefer to be one that worked with dashboard
(unless there is a better recomendation out there??).

1b)
I have also been debating moving the puppetdb (that takes a ton of
resources on the system) to a second server and just letting the first
server be the puppetmaster/ca/ect. Thoughts on that? Any easier? Harder? I
figure it would be a lot easier to configure the balance, but I am not sure
what the consequences are or if it is even a good idea at all.

2)
I read in the changelog that the older clients should work with the
backward compatibility, but is it worth even trying? Or should I just push
out the updated puppet client wherever I can before adding the server into
my new build? (not hard to do in my environment)

3)
Any got-cha!'s that I should be aware of? Any suggestions to make this
process smoother? Any recommendations for a big jump (more like complete
replacement) like this?

Thanks!
~Stack~