I have a few tips after watching the game. I need to follow a few of these myself. You probably know most of these but they were all small problems I saw. A few small problems can equal big problems. You definitely did some things well and early vill production looked good to me.
1- Keep your explorers moving/collecting treasures at all times. Scout his base multiple times and scout for enemy forward before/while you think he may be transitioning.
2- Put Villager collecting crates to work immediately after he/she is done collecting. That's valuable seconds off your age up time.
3- Always send buffalo to base immediately and use the edge of the map if you think the enemy may be scouting the area. Use way-points for this. They can also help you scout on the way if you know where the enemy explorer is.
4- Scout ahead before dropping your military rickshaw. Sometimes dropping it in a direct path to the enemy base is a bad idea and sometimes they will look for players to drop theirs in a direct line from base-to-base. Put it off to the side. Make him scout to find it (too late hopefully).
5- Scout the enemy base well and often to find where resources/vills are so that when/if you raid you can use the best path for surprise. Keep an eye on explorers when near the base. You don't want to give your enemy valuable XP by allowing him to kill your explorer/explorers.
6- Attacking outer houses (away from main base) is better than houses withing TC fire range so early. But it's even better to look for villies instead and kill or make them garrison. Try to mess up his early economy as much as possible. Then back away, when they un-garrison, do it again. Villies before houses if you can.
7- Only garrison vills that are in danger at your base when raided. If he heads out to attack vills on the other side, garrison them and ungarrison the others to the safe side. Try to keep as many vills working as much as possible without getting them killed. Then
always be sure to un-garrison ASAP.
8- A fews short walls within your base would have made his hussar raiding less effective and protected your vills better. You would have also been able to kill more cav if he committed and the walls held him up. Don't wall in, just a few walls to hold him up, give you more warning and allow vills a good escape route.
9- Keep your army active. Idle army is an army waiting to lose a battle sometimes. Harass, scout and kill.
10- You went after his LI with your cav mostly. Take out the organs guns/cannon first (use FOW whenever you can, come around back or in from a side to attack cannons) and then attack the LI. Your LI should have been moved back from organs to a point where he has to move them to engage LI while your cav attacks the guns/cannons and while your LI got in position to attack his LI out of range of his guns.
11- Going after Port TC's early is a bad idea when the TC is under cover fire from another TC. It's better to attack/harass vills.
12- Sometimes retreating is good so that you can get the counters you need to beat the army you just retreated from.
I stopped villie production earlier then I should have but still a vast improvement in my opinion.
NEVER stop. If you have 5 military unit queried but not enough for a villager, un-query a military unit/units, query a villager and then query the final military unit/units before they spawn if you can.
GG
I know these are just general rules and that you probably know them all but I don't play Japan so I can't give you any civ-specific tips. I'll leave that up to the guys that know.