What exactly is slowing down the game?
Each object on your farm has a "digital footprint" that contributes to the loading overhead. As general guidelines, the following philosophies are helpful:
1) Have all trees/animals in their buildings. Trees and animals outside will definitely impact load times, especially those calves/foals.
2) Decorations are okay, but if you have uncompleted buildings (think seed spitters and other items with parts), then that impacts loading times. The backend data of what's completed and how many additional parts are required to complete a given building will impact load times.
3) Yes, trees in orchards make a HUGE different. I have 400+ orchards on Winter Wonderful and that's all I have and loading times are horrible!
4) Fences, haybales surely surely surely have an impact. I don't have a single fence on any of my 23 farms.
5) Mastery signs MUST go into their billboards or into storage. They also have quite a bit of footprint.
6) The number of crops you are planting is not a big deal (unless you use turbos -- then that kinda slows your farm down just until the turbo has done its job).
7) I recently expanded my EC to 58x58 and the sheer number of plots is causing my farm to load slowly. So assuming you have huge farms, it is best not to plow the full space like I did.
8) Put your vehicles in a garage, open all your eggs/babies or delete them, put all blooms in a bloom orchard, etc.
Bottomline -- the less items out in the open the better. Farm Clutter = Loading Sputter. Hope this helps!
Edit: I must also add that any other flash process on your computer will slow down the game. YouTube, Google Docs, Dirt Farmer. etc. It is best to install Adblock Plus for a cleaner/smoother Farmville experience. If you post a picture of your farms... I can give you more streamlined advice.