When I was a teenager my mother's 1981 Toyota Celica developed this problem. I already had the habit of pulling the key out as I turned it back, so sometimes it would not be fully off and I wouldn't know it (accessory position). It was like this once when I jokingly tried other keys in it and I was shocked when it started with a completely different key! I then started it with a spoon just for fun.
