I installed Apache, no problem. I installed php, no problem. php updated Apache's httpd.conf as expected. However, when I restarted Apache:
The requested operation has failed!
And here I thought Microsoft gave unhelpful error messages. Some time Googling later, I found this post which seemed to be much the same issue.
So I uninstalled the php msi, downloaded the same file in zip format, and extracted to the same folder (in my case, C:\PHP). Restarted Apache and there were no issues.