I found an apparently outdated arduino boot cloner sketch, originally designed for an atmel atmega8 chip here. I also downloaded the ADABoot, tried to get the most recent one, it was ADABootLoaderR3_9D15.zip, and had the hex files included so I didn't have to compile those.
I edited the PDE and converted it to a multi-load configuration with DIP switch control (168,368,644P) with proper fuse settings and three different bootloaders in PROGMEM tables. The original had been quite space-constrained as it was designed to load on a atmega8, which had very little program flash. I'd add more chips, but those are all I have. Besides, I the more advanced ones would require a completely different type of socket, which I don't have plans to get any time soon.
I went ahead and used the arduino IDE to compile and load, figuring that I wouldn't have to put up with it for long, but I was wrong. Stupid thing kept screwing up too, but I got that figured out.
So I got it to load successfully, and it burned the new bootloader (the right one too, I made it tell me on the serial console which one it was loading) and swapped the atmega168 on my diecimila (which has a busted pin) with one of the two I just burned. No joy, didn't work at all. I have no clue why. Didn't even flash the debug LED at me!
So I swapped them back and tried again. Only now I can't get the chip to talk back to the boot cloner at all! Not even the basic "program enable" command gets a response back now. The fact that it indicated that it burned successfully means that the chip was talking back to the programmer, but it isn't anymore. Maybe I've screwed something up in the software? I've tried any number of things to get it going again at this point, so it's possible.
I've even completely torn the breadboarded circuit apart and put it back together again with the diecimila on the other side of the board because the wiring worked better that way. I've tried the other atmega168 I have. I've added more debug statements to the code. I've added a little delay when pulsing SCK. I've vastly improved error handling and reporting. I've added an additional delay and start button released checking to the error reporting so it doesn't cycle through to burning again instantly. So I know exactly what the problem is: The chip to be burned doesn't talk back. At all. No clue why.
This is the part of this stuff that I hate. I have no real idea what to try next, so anything I try will be random. (Actually, I've done quite a few mostly random things already, as you may have guessed) I've wracked my brains to no avail. I've spent hours on this problem. I've read up on it. I've consulted the datasheet. I've checked and recheck the pinouts, schematic, and wiring connections. Why doesn't it work? I have no idea. It's so simple, yet it has obviously gone wrong somewhere.
I hope that I will soon move on to the next step: triumphant success. If I only knew how to get there from here.
So I write a blog post, and in the process a few things have occured to me:
Go back to the original version of the software, doing it up right this time with a makefile, bootloaders in separate source files, etc.
Run another pin over to TOSC1 (or is it TOSC2?) and put a PWM on it on the assumption that the clock source fuses are screwed up.
I think I'll try the software first, as I'm more comfortable with software anyway. I do wish I'd bought the atmega368 chip that I thought I had instead of the two atmega168's that I actually did though.